Universal Ink Data Format

The Universal Ink Model is serialized as Protocol Buffers v3 message and wrapped within a RIFF container. The sections below describe the container and the protobuf serialization scheme.

RIFF Container

The Universal Ink Model (UIM) is serialized into a RIFF container, within the RIFF chunk UIM3. This chunk contains two sub-chunks, namely HEAD and DATA. The HEAD chunk contains the current version of the specification. The version is encoded using 3 bytes and is defined as follows: [major-version]-[minor-version]-[patch-version].

The ink model is serialized as Protocol Buffers message as described in section 8.2 Protocol Buffers Serialization Scheme and stored in the DATA chunk.

Figure 1: RIFF diagram

Ink Tree

The ink tree is built with a generic node structure. For building the tree the depth attribute reflects the depth within the tree. For serialization of the tree structure, the depth first pre-order tree serialization is applied, as illustrated in Figure 3.

Each node has an unique identifier id which is relevant for the semantic statements as an identifier for the subject. The index attribute links to the path or sensor data index. The groupBoundingBox is optional and assists with easier visual debugging or to highlight the relevant area for clickable options. Only when the type of the node is CHUNK the attributes chunkFromIndex and chunkToIndex are relevant for sub-indexing.

Figure 2: Depth-First tree order in red: F, B, A, D, C, E, G, I, H.

Data compression

Encoding

The encoding process works as follows:

Step 1. (optional) Convert floating-point values to integer values using externally-specified decimal precision:

integerValue = floatValue * (10 ^ decimalPrecision);

Note: Step 1 is only applied if there the datatype is not an integer datatype (e.g., UINT32, UINT64, INT32, or INT64).

Step 2. Perform delta encoding on the converted values:

encodedValues[0] = integerValues[0];
for(i = 1; i < n; i++)
{
    encodedValues[i] = integerValues[i] - integerValues[i - 1];
}

Note: In converting between floating-point values and fixed-point (integer) values, some precision can be lost. If precision is critical to application functionality, you must take this into consideration. A possible solution is to rely on IEEE 754: in this case, for every value to be converted, you should use the appropriate decimalPrecision and an approximation that remains unchanged during this conversion.

Decoding

To restore the original values, the encoding process is reversed, as follows:

Step 1. Perform delta decoding to restore the integer values:

integerValues[0] = encodedValues[0];
for(i = 1; i < n; i++)
{
    integerValues[i] = integerValues[i - 1] + encodedValues[i];
}

Step 2. (optional) Convert integer values to floating-point values using externally-specified decimal precision:

floatValue = integerValue / (10 ^ decimalPrecision)

Note: Step 2 is only applied if the datatype is not an integer datatype.

Protobuf Serialization Scheme

BrushPrototype

Definition of polygon prototype for brush.

Field	Type	Label	Description
coordX	float	repeated	List of x coordinates of brush prototype.
coordY	float	repeated	List of y coordinates of brush prototype.
coordZ	float	repeated	List of z coordinates of brush prototype [for 3D rendering].
indices	uint32	repeated	List of indices of brush prototype [for 3D rendering].
shapeURI	string	optional	URI uniquely identifying shape's polygon.
size	float	optional	Defines prototype per path point size.

Brushes

Brush descriptions, needed for ink rasterization.

Field	Type	Label	Description
vectorBrushes	VectorBrush	repeated	List of defined vector brushes.
rasterBrushes	RasterBrush	repeated	List of defined raster brushes.

ChannelData

List of data items.

Field	Type	Label	Description
sensorChannelID	string	optional	Referencing InkSensorChannel via id.
values	sint32	repeated	Sample values delta encoded with provided precision.

Environment

Environment defines the all details regarding the input environment:

• Operating System: name, version, etc.
• Application: name, version, etc.
• Geographic information: GPS coordinates.

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
properties	Property	repeated	Environment properties, such as OS name, OS, version, etc.

Float32

Float value wrapper to avoid cross-platform zero value issues.

Field	Type	Label	Description
value	float	optional	Float value wrapped within the message.

InkData

Contains all ink data.

Field	Type	Label	Description
strokes	Stroke	repeated	Data holder for strokes.

InkInputProvider

Definition of input provider types.

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
type	InkInputProviderType	optional	Type of used hardware - PEN, TOUCH, MOUSE, or controller.
properties	Property	repeated	InkInputProvider properties, such as PenID.

InkObject

Ink Object as base structure to capture all relevant data.

• Sensor data recorded from the input sensor
• Brush configurations defining the different brush types used
• Ink data with the visual geometry of the ink strokes
• Ink tree structuring the ink data
• Meta-data describing the author, document, etc.
• Knowledge graph with semantics

Field	Type	Label	Description
inputData	InputData	optional	Data holder for ink input related data.
inkData	InkData	optional	Data holder for ink related ink data.
brushes	Brushes	optional	Rasterization tools.
inkTree	Node	repeated	The ink-tree: this is the ink-centric data model.
views	View	repeated	A list of views of the ink-tree based on the knowledge graph.
knowledgeGraph	TripleStore	optional	The Knowledge Graph.
transform	Matrix4	optional	Transform is an affine transformation matrix which is applied to all paths.
properties	Property	repeated	Document related properties, such as DOCUMENT, AUTHOR, LAST_MODIFIED, etc.

InputContext

Capturing context of the ink device with reference to the Environment and the SensorContext.

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
environmentID	string	optional	Reference to the Environment.
sensorContextID	string	optional	Reference to the SensorContext.

InputContextData

Container structure for context and input defines.

Field	Type	Label	Description
inputContexts	InputContext	repeated	List of input contexts.
inkInputProviders	InkInputProvider	repeated	List of input providers.
inputDevices	InputDevice	repeated	List of input devices.
environments	Environment	repeated	List of environment setups.
sensorContexts	SensorContext	repeated	List of sensor contexts.

InputData

Input Data bundles the input.

Field	Type	Label	Description
inputContextData	InputContextData	optional	Input context data.
sensorData	SensorData	repeated	Data holder for sensor data.

InputDevice

InputDevice with its properties.

The properties can contain:

• Communication Protocol: USB, BTC, BLE, SPP, WIFI
• Communication id: version-WILL3-VID, PID; MAC; UID; COM_PORT
• Device Name: Wacom Intuos Pro M, Apple iPad 8, Samsung GalaxyTab 10
• Serial number
• Firmware Version (MCU)
• Secondary Firmware Version (BT, WIFI) - different modules provides version for itself
• Orientation: PORTRAIT, LANDSCAPE, PORTRAIT_REVERSE, LANDSCAPE_REVERSE or 0, 90, 180, 270
• Sensor size

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
properties	Property	repeated	InputDevice properties.

Matrix4

Representation of a 4x4 affine matrix

| m00 m01 m02 m03 | | m10 m11 m12 m13 | | m20 m21 m22 m23 | | m30 m31 m32 m33 |

Field	Type	Label	Description
m00	float	optional
m01	float	optional
m02	float	optional
m03	float	optional
m10	float	optional
m11	float	optional
m12	float	optional
m13	float	optional
m20	float	optional
m21	float	optional
m22	float	optional
m23	float	optional
m30	float	optional
m31	float	optional
m32	float	optional
m33	float	optional

Node

The node message used for the definition of tree-structures.

Field	Type	Label	Description
id	string	optional	Node unique identifier
depth	uint32	optional	Depth in hierarchy structure
index	uint32	optional	Index in InkData.paths or in InkData.sensorData data holder
type	NodeType	optional	(PATH & SENSOR_DATA nodes only)Node type
groupBoundingBox	Rectangle	optional	Bounding box (Group nodes only)
chunkFromIndex	uint32	optional	Chunk beginning (CHUNK node only)
chunkToIndex	uint32	optional	Chunk ending (CHUNK node only)

PathPointProperties

Pipeline properties which not varies. Variable ones are serialized via Stroke message.

Field	Type	Label	Description
size	Float32	optional	Brush size.
red	Float32	optional	Color value [0, 1] for red channel.
green	Float32	optional	Color value [0, 1] for green channel.
blue	Float32	optional	Color value [0, 1] for blue channel.
alpha	Float32	optional	Color value [0, 1] for alpha channel.
rotation	Float32	optional	Brush rotation z.
scaleX	Float32	optional	Brush scale x.
scaleY	Float32	optional	Brush scale y.
scaleZ	Float32	optional	Brush scale z [for 3D rendering].
offsetX	Float32	optional	Brush offset x.
offsetY	Float32	optional	Brush offset y.
offsetZ	Float32	optional	Brush offset z [for 3D rendering].

Property

Encodes a single property

Field	Type	Label	Description
name	string	optional	Name of the property.
value	string	optional	Value of property.

RasterBrush

A brush defined with raster images

Field	Type	Label	Description
name	string	optional	the brush descriptor.
spacing	float	optional	Distance between neighbour particles.
scattering	float	optional	The scattering along the curve normal.
rotationMode	RotationMode	optional	The particle rotation mode of the brush.
shapeTexture	bytes	repeated	List of png images that contains the shape texture.
shapeTextureURI	string	repeated	List of URIs associated to the shape texture.
fillTexture	bytes	optional	List of png image that contains the fill texture.
fillTextureURI	string	optional	URI identifying the fillTexture.
fillWidth	float	optional	Width of the fill tile.
fillHeight	float	optional	Height of the fill tile.
randomizeFill	bool	optional	Specifies whether the fill texture is randomly displaced.
blendMode	BlendMode	optional	The applied blend mode.

Rectangle

Representation of rectangle

Field	Type	Label	Description
x	float	optional	Lower-left x-coordinate point of rectangle.
y	float	optional	Lower-left y-coordinate point of rectangle.
width	float	optional	Width of rectangle.
height	float	optional	Height of rectangle.

SemanticTriple

A semantic triple, or simply triple, is the atomic data entity data model. As its name indicates, a triple is a set of three entities that codifies a statement about semantic data in the form of subject predicate object expressions.

Field	Type	Label	Description
subject	string	optional
predicate	string	optional
object	string	optional

SensorChannel

Information about the sensor channel.

Supported channel types:

• will:///input/3.0/channel/X
• will:///input/3.0/channel/Y
• will:///input/3.0/channel/Z
• will:///input/3.0/channel/Timestamp
• will:///input/3.0/channel/Pressure
• will:///input/3.0/channel/RadiusX
• will:///input/3.0/channel/RadiusY
• will:///input/3.0/channel/Altitude
• will:///input/3.0/channel/Azimuth
• will:///input/3.0/channel/Rotation

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
type	string	optional	URI which describes uniquely this channel. Supported types are:
metric	InkSensorMetricType	optional	User defined channels which introduce custom meaning should be provided from user side.Indicates metric used in calculating the resolution for the data item.
resolution	double	optional	Is a decimal number giving the number of data item increments.
min	float	optional	Per physical unit., e.g. if the physical unit is in m and device units./ Resolution is 100000, then the value 150 would be 0.0015 m.Minimal value of the channel
max	float	optional	Maximal value of the channel
precision	uint32	optional	Precision of integer encoding, needed for encoded float values

SensorChannelsContext

The SensorChannelsContext groups sensor channels that are producing data with the same sampling rate.

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
channels	SensorChannel	repeated	Channels in this group provides aligned values in the same time.
samplingRateHint	Uint32	optional	Hint for the intended sampling rate of the channel [Optional].
latency	Uint32	optional	Latency measure in milliseconds [Optional].
inkInputProviderID	string	optional	Reference to the InkInputProvider.
inputDeviceID	string	optional	Reference to the InputDevice.

SensorContext

Each input device has a SensorContext describing the available sensors of a device. One file can contain Ink data from two device of the same type with a shared context.

Field	Type	Label	Description
id	string	optional	MD5 hash according to the MD5 hash-based Unique Identifier Generation Scheme.
sensorChannelsContext	SensorChannelsContext	repeated	Grouping context for device sensor channels.

SensorData

SensorData is the central data structure for storing raw data from the ink sensor.

Field	Type	Label	Description
id	string	optional
inputContextID	string	optional	References the InputContext via id.
state	InkState	optional	State of the ink.
timestamp	uint64	optional	Timestamp for first sample of the stroke, measured in milliseconds.
dataChannels	ChannelData	repeated	(time when first X & Y are reported). Data channels encoding the data list.

Stroke

Visual path structure. Stroke in its simplest form contains one or more Segments (Catmull-Rom), which includes a series of positional points, Positions. In addition to those properties, digital ink strokes include start parameter and end parameter as Stroke entity's properties.

Field	Type	Label	Description
id	string	optional	Internal id.
startParameter	float	optional	Spline start parameter [0, 1].
endParameter	float	optional	Spline end parameter [0, 1].
splineX	float	repeated	Spline values for x.
splineY	float	repeated	Spline values for y.
splineZ	float	repeated	Spline values for z [for 3D rendering].
red	float	repeated	Color values [0, 1] for red channel.
green	float	repeated	Color values [0, 1] for green channel.
blue	float	repeated	Color values [0, 1] for blue channel.
alpha	float	repeated	Color values [0, 1] for alpha channel.
size	float	repeated	Brush size.
rotation	float	repeated	Brush rotation z.
scaleX	float	repeated	Brush scale x.
scaleY	float	repeated	Brush scale y.
scaleZ	float	repeated	Brush scale z [for 3D rendering].
offsetX	float	repeated	Brush offset x.
offsetY	float	repeated	Brush offset y.
offsetZ	float	repeated	Brush offset z [for 3D rendering].
sensorDataOffset	uint32	optional	Index of points mapping between raw and processed paths.
sensorDataID	string	optional	Reference sensor data.
sensorDataMapping	uint32	repeated	Explicit mapping between indices of Stroke and SensorData, used when input rate is very high and provides unwanted points.
style	Style	optional	Style that is applied to the path.

Style

A style message, used for Stroke customization.

Field	Type	Label	Description
properties	PathPointProperties	optional	Static values of properties which do not exist per per path point
brushURI	string	optional	Reference to Brush used for stroke rasterization.
particlesRandomSeed	uint32	optional	Particles random seed, required for particle strokes.
renderModeURI	string	optional	Defines additional information about stroke visualisation, such as ERASER.

TripleStore

Encapsulates a list of TripleGroups.

Field	Type	Label	Description
statements	SemanticTriple	repeated	List of semantic triples.

Uint32

Unsigned integer value wrapper to avoid cross-platform zero value issues.

Field	Type	Label	Description
value	uint32	optional	Unsigned integer value wrapped within the message.

VectorBrush

A brush defined with a polygon (polyhedron for 3D).

Field	Type	Label	Description
name	string	optional	Brush descriptor.
prototype	BrushPrototype	repeated	Polygon prototype for the brush.
spacing	float	optional	Spacing value.

View

A message representing a view of the ink-tree, which is serialized in form of another tree.

Field	Type	Label	Description
name	string	optional	View descriptor URI (unique in the model scope).
tree	Node	repeated	The tree structure of the current view.

BlendMode

The blend mode enum lists the different blend modes which can be applied to raster brushes.

Name	Number	Description
SOURCE_OVER	0	This is the default setting and draws new shapes on top of the existing canvas content. Also known as NORMAL.
DESTINATION_OVER	1	New shapes are drawn behind the existing canvas content. Also known as NORMAL_REVERSE.
DESTINATION_OUT	2	The existing content is kept where it doesn't overlap the new shape. Also known as ERASE.
LIGHTER	3	Where both shapes overlap the color is determined by adding color values. Also known as ADD.
COPY	4	Only the new shape is shown. Also known as NONE.
MIN	5	The result is the minimum of both color. The result is a darker color.
MAX	6	The result is the maximum of both color. The result is a lighter color.

InkInputProviderType

Defining the different types of input.

Name	Number	Description
PEN	0	Stylus, smart pens, pen displays, signature capturing device, etc.
TOUCH	1	Touch controller input: Finger or passive stylus.
MOUSE	2	Mouse device.
CONTROLLER	3	3-DOF or 6-DOF input devices.

InkSensorMetricType

Supported metric types

Name	Number	Description
LENGTH	0	Underlying si unit is meter.
TIME	1	Underlying si unit is second.
FORCE	2	Underlying si unit is newton.
ANGLE	3	Underlying si unit is radian.
NORMALIZED	4	Percentage, expressed as a fraction (1.0 = 100%) relative to max-min.
LOGICAL	5	Logical value, true / false values.
DIMENSIONLESS	6	No metric defined for sensor.

InkState

The ink device state defines the state of the Ink device. WILL 3 supports different modes:

• Writing on a plane
• Hovering above a surface
• Moving in air (VR/AR/MR) interaction
• Only hovering in the air

Name	Number	Description
PLANE	0	Ink device is writing on a surface.
HOVERING	1	Hovering over a surface.
IN_VOLUME	2	Using the ink device in the air, with active inking.
VOLUME_HOVERING	3	Moving the pen in the air with disabled inking.

NodeType

Defines the node type.

Name	Number	Description
PATH_GROUP	0	PATH_GROUP is hierarchically combining stroke nodes.
SENSOR_DATA_GROUP	1	SENSOR_DATA_GROUP is hierarchically combining sensor data nodes.
PATH	2	PATH is a node type where nodes directly point to a stroke.
SENSOR_DATA	3	SENSOR_DATA is a node type where nodes directly point to a sensor data sample sequence.

RotationMode

The rotation parameter specifies the rotation of the particle.

Name	Number	Description
NONE	0	Indicates that the shape will not be rotated.
RANDOM	1	Indicates that the shape will be rotated randomly before it renders.
TRAJECTORY	2	Indicates that the shape will be rotated to match the path trajectory.

Scalar Value Types

.proto Type	Notes	C++ Type	Java Type	Python Type
double		double	double	float
float		float	float	float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers - if your field is likely to have negative values, use sint32 instead.	int32	int	int
int64	Uses variable-length encoding. Inefficient for encoding negative numbers - if your field is likely to have negative values, use sint64 instead.	int64	long	int/long
uint32	Uses variable-length encoding.	uint32	int	int/long
uint64	Uses variable-length encoding.	uint64	long	int/long
sint32	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int32s.	int32	int	int
sint64	Uses variable-length encoding. Signed int value. These more efficiently encode negative numbers than regular int64s.	int64	long	int/long
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	int
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	int/long
sfixed32	Always four bytes.	int32	int	int
sfixed64	Always eight bytes.	int64	long	int/long
bool		bool	boolean	boolean
string	A string must always contain UTF-8 encoded or 7-bit ASCII text.	string	String	str/unicode
bytes	May contain any arbitrary sequence of bytes.	string	ByteString	str