Ink Geometry Pipeline & Rendering
The different rendering modes have already been introduced. This guide explains how to setup the pipeline and the rendering, step-by-step.
The visual result is dependent on the brush configuration, the pipeline configuration and the configuration of the platform rendering.
Raster Ink
This section describes the best practice for setting up the geometry and rendering pipeline for Raster Ink using a step-by-step approach.
Brush configuration
To setup the raster particle brush, two textures are needed:
The shape and the fill texture:
The shape parameter specifies the shape of the particle.
It accepts an image in which the non-transparent pixels define the particle shape.
The fill parameter specifies the fill pattern of the shape.
It accepts an image from which the pattern is sampled.
For both textures, you can also pass a sequence of images in this parameter. In this case, the method interprets them as mipmaps.
...
fun pencil(context: Context): RasterBrush {
val opts = BitmapFactory.Options()
opts.inSampleSize = 1
opts.inScaled = false
// Texture for shape
val shapeTexture =
BitmapFactory.decodeResource(context.resources, R.drawable.essential_shape, opts)
// Texture for fill
val fillTexture =
BitmapFactory.decodeResource(context.resources, R.drawable.essential_fill_11, opts)
// Convert to PNG
val stream = ByteArrayOutputStream()
shapeTexture!!.compress(Bitmap.CompressFormat.PNG, 100, stream)
val shapeTextureByteArray = stream.toByteArray()
val stream2 = ByteArrayOutputStream()
fillTexture!!.compress(Bitmap.CompressFormat.PNG, 100, stream2)
val fillTextureByteArray = stream2.toByteArray()
// Create the raster brush
var brush = RasterBrush(
URIBuilder.getBrushURI("raster", "Pencil"), // name of the brush
0.15f, // spacing
0.15f, // scattering
RotationMode.RANDOM, // rotation mode
listOf(shapeTextureByteArray), // shape texture
listOf(), fillTextureByteArray, // fill texture
"", // fill texture URI
fillTexture.width.toFloat(), // width of texture
fillTexture.height.toFloat(), // height of texture
false, // randomized fill
BlendMode.MAX // mode of blending
)
shapeTexture.recycle()
fillTexture.recycle()
return brush
}
...
public PencilTool(Graphics graphics)
{
Fill = new PixelInfo(new Uri("ms-appx:///Assets/textures/essential_fill_11.png"));
Shape = new PixelInfo(new Uri("ms-appx:///Assets/textures/essential_shape.png"));
Brush.Scattering = 0.05f;
Brush.RotationMode = ParticleRotationMode.RotateRandom;
Brush.FillTileSize = new Size(32.0f, 32.0f);
Brush.FillTexture = graphics.CreateTexture(Fill.PixelData);
Brush.ShapeTexture = graphics.CreateTexture(Shape.PixelData);
ParticleSpacing = 0.3f;
}
/* **************** RASTER BRUSH configuration **************** */
pencil: new BrushGL(
URIBuilder.getBrushURI("raster", "Pencil"),
"/images/textures/essential_shape.png",
"/images/textures/essential_fill_11.png",
{
spacing: 0.15,
scattering: 0.15
}),
...
PathPoint calculators
With all the relevant brushes defined, we need to configure the PathPoint calculators which influence the way sensor data changes the width, intensity and the tilt of the ink strokes.
First, the layout needs to be configured. Depending on the input provider, the layout can be adapted. For instance for touch input you are not receiving tilt nor pressure information.
...
import com.wacom.ink.PathPointLayout
...
override fun getLayout(): PathPointLayout {
// Define different layouts for stylus and touch input
if (isStylus) {
return PathPointLayout(
PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.SIZE,
PathPoint.Property.ALPHA,
PathPoint.Property.ROTATION,
PathPoint.Property.OFFSET_X
)
} else {
return PathPointLayout(
PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.SIZE,
PathPoint.Property.ALPHA
)
}
}
...
...
using Wacom.Ink.Geometry;
using Wacom.Ink.Rendering;
...
public override PathPointLayout GetLayout(Windows.Devices.Input.PointerDeviceType deviceType)
{
switch (deviceType)
{
case Windows.Devices.Input.PointerDeviceType.Mouse:
case Windows.Devices.Input.PointerDeviceType.Touch:
return new PathPointLayout(PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.Size,
PathPoint.Property.Alpha);
case Windows.Devices.Input.PointerDeviceType.Pen:
return new PathPointLayout(PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.Size,
PathPoint.Property.Alpha,
PathPoint.Property.OffsetX,
PathPoint.Property.OffsetY);
default:
throw new Exception("Unknown input device type");
}
}
// In the JavaScript implementation layout is predefined within PathPointContext
let context = new PathPointContext();
As touch and stylus input can be optimized for the different kinds of input sensor data, you need to define calculators which receive the previous, current and next sensor data input and computes the appropriate PathPoint.
For instance, touch data includes neither pressure information nor tilt orientation like a stylus.
Thus, to achieve the same difference in alpha intensity or variation of ink stroke width, only speed information can be used.
The function to compute width based on the current speed of the input data is offered by the engine.
Here, the min and max value define the range of the width, while min and max speed define how the width is changed within its range. The result is a normalized value between 0 and 1. Finally, with the remap function the normalized value can be modified, e.g., to reverse the behavior. So that, for instance a slow movement can increase the width of the stroke rather reduce it, while if the finger moves faster the stroke becomes thinner.
...
companion object {
val uri = URIBuilder.getToolURI("raster", "pencil")
// Minimum size of the pencil tip
val MIN_PENCIL_SIZE = 4f
// Maximum size of the pencil tip
val MAX_PENCIL_SIZE = 25f
// Minimum alpha values for the particles
val MIN_ALPHA = 0.1f
// Maximum alpha values for the particles
val MAX_ALPHA = 0.7f
// Unit for speed is px/second.
// NOTE: This needs to be optimized for different Pixel densities of different devices
val MAX_SPEED = 15000f
}
...
override val touchCalculator: Calculator = { previous, current, next ->
// Use the following to compute size based on speed:
var size = current.computeValueBasedOnSpeed(
previous,
next,
minValue = MIN_PENCIL_SIZE,
maxValue = MAX_PENCIL_SIZE,
minSpeed = 80f,
maxSpeed = MAX_SPEED,
// reverse behaviour
remap = { 1f - it}
)!!
var alpha = current.computeValueBasedOnSpeed(
previous,
next,
minValue = MIN_ALPHA,
maxValue = MAX_ALPHA,
minSpeed = 80f,
maxSpeed = MAX_SPEED,
// reverse behaviour
remap = { 1f - it}
)
if (alpha == null) {
alpha = previousAlpha
} else {
previousAlpha = alpha
}
PathPoint(current.x, current.y, size = size, alpha = alpha)
}
...
private static readonly ToolConfig mSizeConfig = new ToolConfig()
{
initValue = 4,
finalValue = 10,
minValue = 4,
maxValue = 10,
minSpeed = 80,
maxSpeed = 1400,
};
private static readonly ToolConfig mAlphaConfig = new ToolConfig()
{
initValue = 0.1f,
finalValue = 0.5f,
minValue = 0.1f,
maxValue = 0.5f,
minSpeed = 80,
maxSpeed = 1400,
};
...
protected override ToolConfig SizeConfig => mSizeConfig;
protected override ToolConfig AlphaConfig => mAlphaConfig;
...
/// <summary>
/// Calculator delegate for input from mouse input
/// Calculates the path point properties based on pointer input.
/// </summary>
/// <param name="previous"></param>
/// <param name="current"></param>
/// <param name="next"></param>
/// <returns>PathPoint with calculated properties</returns>
protected PathPoint CalculatorForMouseAndTouch(PointerData previous, PointerData current, PointerData next)
{
var size = current.ComputeValueBasedOnSpeed(previous, next,
SizeConfig.minValue, SizeConfig.maxValue,
SizeConfig.initValue, SizeConfig.finalValue,
SizeConfig.minSpeed, SizeConfig.maxSpeed,
SizeConfig.remap);
if (size.HasValue)
{
PreviousSize = size.Value;
}
else
{
size = PreviousSize;
}
var alpha = current.ComputeValueBasedOnSpeed(previous, next,
AlphaConfig.minValue, AlphaConfig.maxValue,
AlphaConfig.initValue, AlphaConfig.finalValue,
AlphaConfig.minSpeed, AlphaConfig.maxSpeed,
AlphaConfig.remap);
if (alpha.HasValue)
{
PreviousAlpha = alpha.Value;
}
else
{
alpha = PreviousAlpha;
}
PathPoint pp = new PathPoint(current.X, current.Y)
{
Size = size,
Alpha = alpha
};
return pp;
}
// In the JavaScript implementation you need to define the dynamics for the PathPoint calculators
/* ******* RASTER TOOLS ******* */
pencil: {
brush: BrushPalette.pencil,
dynamics: {
size: {
value: {
min: 4,
max: 5
},
velocity: {
min: 80,
max: 1400,
remap: v => ValueTransformer.reverse(v)
}
},
alpha: {
value: {
min: 0.05,
max: 0.2
},
velocity: {
min: 80,
max: 1400
}
},
rotation: {
dependencies: [SensorChannel.Type.ROTATION, SensorChannel.Type.AZIMUTH]
},
scaleX: {
dependencies: [SensorChannel.Type.RADIUS_X, SensorChannel.Type.ALTITUDE],
value: {
min: 1,
max: 3
}
},
scaleY: {
dependencies: [SensorChannel.Type.RADIUS_Y],
value: {
min: 1,
max: 3
}
},
offsetX: {
dependencies: [SensorChannel.Type.ALTITUDE],
value: {
min: 2,
max: 5
}
}
}
}
The configuration for stylus input offers more opportunities to configure a natural behavior of a real world writing tool. Here, pressure sensor data can be used to increase the width of tools or change the intensity of the color. Also, if a stylus is tilted it would naturally change the contact surface area. If you want to implement pencil hatching, water color effects, or fountain pens, you need to implement the effect here.
NOTE: Input sensor quality varies, thus, test if the desired behavior is achieved with different kinds of stylus input.
...
var previousAlpha = 0.2f
...
override val stylusCalculator: Calculator = { previous, current, next ->
// calculate the offset of the pencil tip due to tilted position
val cosAltitudeAngle = cos(current.altitudeAngle!!)
val sinAzimuthAngle = sin(current.azimuthAngle!!)
val cosAzimuthAngle = cos(current.azimuthAngle!!)
val x = sinAzimuthAngle * cosAltitudeAngle
val y = cosAltitudeAngle * cosAzimuthAngle
val offsetY = 5f * -x
val offsetX = 5f * -y
// compute the rotation
val rotation = current.computeNearestAzimuthAngle(previous)
// now, based on the tilt of the pencil the size of the brush size is increasing, as the
// pencil tip is covering a larger area
val size = max(MIN_PENCIL_SIZE,
min(MAX_PENCIL_SIZE, MIN_PENCIL_SIZE + 20f * abs(current.altitudeAngle!!)))
// Change the intensity of alpha value by pressure of speed, if available else use speed
var alpha = if (current.force == -1f) {
current.computeValueBasedOnSpeed(
previous,
next,
minValue = MIN_ALPHA,
maxValue = MAX_ALPHA,
minSpeed = 0f,
maxSpeed = 3500f,
// reverse behaviour
remap = { 1.0f - it}
)
} else {
current.computeValueBasedOnPressure(
minValue = MIN_ALPHA,
maxValue = MAX_ALPHA,
minPressure = 0.0f,
maxPressure = 1.0f,
remap = { v: Float -> v.toDouble().pow(1).toFloat() }
)
}
if (alpha == null) {
alpha = previousAlpha
} else {
previousAlpha = alpha
}
PathPoint(
current.x, current.y,
alpha = alpha, size = size, rotation = rotation,
offsetX = offsetX, offsetY = offsetY
)
}
...
/// <summary>
/// Calculator delegate for input from a stylus (pen)
/// Calculates the path point properties based on pointer input.
/// </summary>
/// <param name="previous"></param>
/// <param name="current"></param>
/// <param name="next"></param>
/// <returns>PathPoint with calculated properties</returns>
protected override PathPoint CalculatorForStylus(PointerData previous, PointerData current, PointerData next)
{
var cosAltitudeAngle = Math.Cos(current.AltitudeAngle.Value);
var sinAzimuthAngle = Math.Sin(current.AzimuthAngle.Value);
var cosAzimuthAngle = Math.Cos(current.AzimuthAngle.Value);
// calculate the offset of the pencil tip due to tilted position
var x = sinAzimuthAngle * cosAltitudeAngle;
var y = cosAltitudeAngle * cosAzimuthAngle;
var offsetY = -x;
var offsetX = -y;
// compute the rotation
var rotation = current.ComputeNearestAzimuthAngle(previous);
// now, based on the tilt of the pencil the size of the brush size is increasing, as the
// pencil tip is covering a larger area
var size = Math.Max(mSizeConfig.minValue, mSizeConfig.minValue + 20f * Math.Abs(current.AltitudeAngle.Value));
// Change the intensity of alpha value by pressure of speed, if available else use speed
var alpha = (!current.Force.HasValue)
? current.ComputeValueBasedOnSpeed(previous, next, 0.1f, 0.7f, null, null, 0f, 3500f)
: ComputeValueBasedOnPressure(current, 0.1f, 0.7f, 0.0f, 1.0f);
if (!alpha.HasValue)
{
alpha = PreviousAlpha;
}
else
{
PreviousAlpha = alpha.Value;
}
PathPoint pp = new PathPoint(current.X, current.Y)
{
Size = size,
Alpha = alpha,
OffsetX = (float)offsetX,
OffsetY = (float)offsetY
};
return pp;
}
Pipeline
The RasterInkBuilder defines the Raster Ink pipeline:
import com.wacom.ink.*
import com.wacom.ink.format.rendering.RasterBrush
import com.wacom.ink.pipeline.PathProducer
import com.wacom.ink.pipeline.SmoothingFilter
import com.wacom.ink.pipeline.SplineInterpolator
import com.wacom.ink.pipeline.SplineProducer
import com.wacom.ink.pipeline.base.ProcessorResult
...
class RasterInkBuilder() {
// The pipeline components:
lateinit var pathProducer: PathProducer
lateinit var smoother: SmoothingFilter
lateinit var splineProducer: SplineProducer
lateinit var splineInterpolator: SplineInterpolator
lateinit var pathSegment: PathSegment
lateinit var tool: RasterTool
...
fun updatePipeline(newTool: Tool) {
if (newTool !is RasterTool) {
throw IllegalArgumentException("Invalid tool")
} else {
tool = newTool
var layout = tool.getLayout()
// Path producer needs to know the layout
pathProducer = PathProducer(layout, tool.getCalculator())
smoother = SmoothingFilter(layout.size) // Dimension
splineProducer = SplineProducer(layout) // Layout
splineInterpolator = SplineInterpolator(
spacing = tool.brush.spacing, // Spacing between two successive sample.
splitCount = 1, // Determines the number of iterations for the
// discretization.
calculateDerivatives = true, // Calculate derivatives
interpolateByLength = true // Interpolate by length
)
pathSegment = PathSegment()
splineProducer.keepAllData = true
}
}
...
// In the JavaScript implementation the setup of the pipeline within InkBuilderAbstract which is inheritat by InkBuilder
this.builder = new InkBuilder();
/// <summary>
/// Manages ink geometry pipeline for raster (particle) brushes.
/// </summary>
public class RasterInkBuilder : InkBuilder
{
#region Fields
private const float defaultSpacing = 0.15f;
private const int splitCount = 1;
#endregion
#region Constructors
public RasterInkBuilder()
{
}
#endregion
#region Properties
public event EventHandler LayoutUpdated;
#endregion
...
#region Public Interface
public void UpdatePipeline(PathPointLayout layout, Calculator calculator, float spacing)
{
bool layoutChanged = false;
bool otherChange = false;
if ((Layout == null) || (layout.ChannelMask != Layout.ChannelMask))
{
Layout = layout;
layoutChanged = true;
}
if (mPathProducer == null || calculator != mPathProducer.PathPointCalculator || layoutChanged)
{
mPathProducer = new PathProducer(Layout, calculator, true);
otherChange = true;
}
if (mSmoothingFilter == null || layoutChanged)
{
mSmoothingFilter = new SmoothingFilter(Layout.Count)
{
KeepAllData = true
};
otherChange = true;
}
if (SplineProducer == null || layoutChanged)
{
SplineProducer = new SplineProducer(Layout, true);
otherChange = true;
}
if (SplineInterpolator == null || layoutChanged)
{
SplineInterpolator = new DistanceBasedInterpolator(Layout, spacing, splitCount, true, true, true);
otherChange = true;
}
((DistanceBasedInterpolator) SplineInterpolator).Spacing = spacing;
if (layoutChanged || otherChange)
{
LayoutUpdated?.Invoke(this, EventArgs.Empty);
}
}
#endregion
}
}
Raster View
Next, setup the raster view:
import com.wacom.ink.egl.EGLRenderingContext
import com.wacom.ink.format.tree.data.Stroke
import com.wacom.ink.format.tree.nodes.StrokeNode
import com.wacom.ink.rasterization.InkCanvas
import com.wacom.ink.rasterization.Layer
import com.wacom.ink.rasterization.StrokeRenderer
import com.wacom.ink.rendering.BlendMode
...
/**
* This is a surface for drawing raster inking.
* Extends from SurfaceView
*/
class RasterView @JvmOverloads constructor(
context: Context,
attrs: AttributeSet? = null,
defStyleAttr: Int = 0
) : SurfaceView(context, attrs, defStyleAttr) {
...
private var inkCanvas: InkCanvas? = null
private lateinit var viewLayer: Layer
private lateinit var strokesLayer: Layer
private lateinit var currentFrameLayer: Layer
private lateinit var strokeRenderer: StrokeRenderer
private var rasterInkBuilder = RasterInkBuilder() // The ink builder
var rasterTool: RasterTool = PencilTool(context) // The default tool
...
init {
rasterInkBuilder.updatePipeline(PencilTool(context))
setZOrderOnTop(true);
holder.setFormat(PixelFormat.TRANSPARENT);
// the drawing is going to be performer on background
// on a surface, so we initialize the surface
holder.addCallback(object : SurfaceHolder.Callback {
override fun surfaceChanged(holder: SurfaceHolder?, format: Int, w: Int, h: Int) {
// here, once the surface is crated, we are going to initialize ink canvas
// firs we check that there is no other inkCanvas, in case there is we dispose it
if (inkCanvas?.isDisposed == false) {
releaseResources();
}
inkCanvas =
InkCanvas(holder, EGLRenderingContext.EGLConfiguration(8, 8, 8, 8, 8, 8))
viewLayer = inkCanvas!!.createViewLayer(w, h)
strokesLayer = inkCanvas!!.createLayer(w, h)
currentFrameLayer = inkCanvas!!.createLayer(w, h)
inkCanvas!!.clearLayer(currentFrameLayer)
strokeRenderer =
StrokeRenderer(inkCanvas, PencilTool(context).brush.toParticleBrush(), w, h)
drawStrokes(strokeNodeList)
renderView()
if (listener != null) {
listener.onSurfaceCreated()
}
}
override fun surfaceDestroyed(holder: SurfaceHolder?) {
releaseResources();
}
override fun surfaceCreated(p0: SurfaceHolder?) {
//we don't have to do anything here
}
})
}
...
/// <summary>
/// Creates and manages rendering brush and graphics objects and interactions between them;
/// Recieves pointer input events and passes them on to the rendering brush to feed into
/// the ink geometry pipeline
/// </summary>
public class Renderer
{
#region Fields
private SwapChainPanel m_swapChainPanel;
private PointerManager m_pointerManager;
private Graphics m_graphics = new Graphics();
#endregion
...
#region Constructor
...
/// <summary>
/// Constructor. Creates rendering brush; initializes graphics
/// </summary>
/// <param name="swapChainPanel">SwapChainPanel on which to render captured ink</param>
/// <param name="brushType">Type of brush to use</param>
/// <param name="thickness">Relative thickness of brush</param>
/// <param name="color">Color of ink</param>
public Renderer(SwapChainPanel swapChainPanel, RasterBrushStyle style, MediaColor color)
{
StrokeHandler = new RasterStrokeHandler(this, style, color, m_graphics);
m_swapChainPanel = swapChainPanel;
m_graphics.GraphicsReady += OnGraphicsReady;
m_graphics.Initialize(m_swapChainPanel, false);
}
#endregion
class InkCanvasRaster extends InkCanvas {
constructor(canvas, width, height) {
super();
this.canvas = InkCanvasGL.createInstance(canvas, width, height);
this.strokesLayer = this.canvas.createLayer();
this.strokeRenderer = new StrokeRendererGL(this.canvas);
}
}
Input Handling
Input events from the operating system are handled using the following:
// This function is going to be call when we touch the surface
fun surfaceTouch(event: MotionEvent) {
// Check the input method
if (event.resolveToolType() == InkInputType.PEN) {
if ((newTool) || (!isStylus)) {
newTool = false
isStylus = true
rasterInkBuilder.updateInputMethod(isStylus)
}
} else {
if ((newTool) || (isStylus)) {
newTool = false
isStylus = false
rasterInkBuilder.updateInputMethod(isStylus)
}
}
for (i in 0 until event.historySize) {
val pointerData = event.historicalToPointerData(i)
rasterInkBuilder.add(pointerData.phase, pointerData, null)
}
val pointerData = event.toPointerData()
rasterInkBuilder.add(pointerData.phase, pointerData, null)
val (added, predicted) = rasterInkBuilder.build()
if (pointerData.phase == Phase.BEGIN) {
// initialize the sensor data each time a new stroke begin
val pair = inkEnvironmentModel.createSensorData(event)
sensorData = pair.first
channelList = pair.second
}
for (channel in channelList) {
when (channel.type) {
InkSensorType.X -> sensorData.add(channel, pointerData.x)
InkSensorType.Y -> sensorData.add(channel, pointerData.y)
InkSensorType.TIMESTAMP -> sensorData.addTimestamp(
channel,
pointerData.timestamp
)
InkSensorType.PRESSURE -> sensorData.add(channel, pointerData.force!!)
InkSensorType.ALTITUDE -> sensorData.add(channel, pointerData.altitudeAngle!!)
InkSensorType.AZIMUTH -> sensorData.add(channel, pointerData.azimuthAngle!!)
}
}
if ((pointerData.phase == Phase.END) && (rasterInkBuilder.splineProducer.allData != null)) {
addStroke()
sensorDataList.add(sensorData)
}
if (added != null) {
drawStroke(event, added, predicted)
}
renderView()
}
/// <summary>
/// Initiates capture of an ink stroke
/// </summary>
private void OnPointerPressed(object sender, PointerRoutedEventArgs args)
{
// If currently there is an unfinished stroke - do not interrupt it
if (!m_pointerManager.OnPressed(args))
return;
m_swapChainPanel.CapturePointer(args.Pointer);
StrokeHandler.OnPressed(m_swapChainPanel, args);
}
/// <summary>
/// Updates current ink stroke with new pointer input
/// </summary>
private void OnPointerMoved(object sender, PointerRoutedEventArgs args)
{
// Ignore events from other pointers
if (!m_pointerManager.OnMoved(args))
return;
StrokeHandler.OnMoved(m_swapChainPanel, args);
}
/// <summary>
/// Completes capture of an ink stroke
/// </summary>
private void OnPointerReleased(object sender, PointerRoutedEventArgs args)
{
// Ignore events from other pointers
if (!m_pointerManager.OnReleased(args))
return;
m_swapChainPanel.ReleasePointerCapture(args.Pointer);
StrokeHandler.OnReleased(m_swapChainPanel, args);
if (!StrokeHandler.IsSelecting)
{
RenderNewStrokeSegment();
BlendCurrentStrokesLayerToAllStrokesLayer();
RenderBackbuffer();
PresentGraphics();
var ptrDevice = args.GetCurrentPoint(m_swapChainPanel).PointerDevice;
StrokeHandler.StoreCurrentStroke(ptrDevice.PointerDeviceType);
}
}
// StrokeHandler for raster strokes
public class RasterStrokeHandler : StrokeHandler
{
...
public override void OnPressed(UIElement uiElement, PointerRoutedEventArgs args)
{
m_startRandomSeed = (uint)mRand.Next();
base.OnPressed(uiElement, args);
ActiveTool.OnPressed(uiElement, args);
}
public override void OnMoved(UIElement uiElement, PointerRoutedEventArgs args)
{
ActiveTool.OnMoved(uiElement, args);
}
public override void OnReleased(UIElement uiElement, PointerRoutedEventArgs args)
{
ActiveTool.OnReleased(uiElement, args);
}
...
}
class InkCanvas extends InkController {
...
begin(sensorPoint) {
if (this.forward) return this.inkCanvasRaster.begin(sensorPoint);
this.reset(sensorPoint);
this.builder.add(sensorPoint);
this.builder.build();
}
move(sensorPoint) {
if (this.forward) return this.inkCanvasRaster.move(sensorPoint);
if (this.requested) {
this.builder.ignore(sensorPoint);
return;
}
this.builder.add(sensorPoint);
if (!this.requested) {
this.requested = true;
this.builder.build();
requestAnimationFrame(() => (this.requested = false));
}
}
end(sensorPoint) {
if (this.forward) return this.inkCanvasRaster.end(sensorPoint);
this.builder.add(sensorPoint);
this.builder.build();
}
...
Finally, draw the stroke:
fun drawStrokes(strokeList: MutableList<Pair<StrokeNode, RasterBrush>>) {
for (stroke in strokeList) {
drawStroke(stroke.first, stroke.second)
}
}
...
// Draw stroke
private fun drawStroke(
event: MotionEvent,
added: InterpolatedSpline,
predicted: InterpolatedSpline?
) {
strokeRenderer.drawPoints(added, defaults, event.action == MotionEvent.ACTION_UP)
if (predicted != null) {
strokeRenderer.drawPrelimPoints(predicted, defaults)
}
if (event.action != MotionEvent.ACTION_UP) {
inkCanvas!!.setTarget(currentFrameLayer, strokeRenderer.strokeUpdatedArea)
inkCanvas!!.clearColor()
inkCanvas!!.drawLayer(strokesLayer, BlendMode.SOURCE_OVER)
strokeRenderer.blendStrokeUpdatedArea(currentFrameLayer, rasterTool.getBlendMode())
} else {
strokeRenderer.blendStroke(strokesLayer, rasterTool.getBlendMode())
inkCanvas!!.setTarget(currentFrameLayer)
inkCanvas!!.clearColor()
inkCanvas!!.drawLayer(strokesLayer, BlendMode.SOURCE_OVER)
}
}
...
public override void RenderAllStrokes(RenderingContext context, IEnumerable<Identifier> excluded, Rect? clipRect)
{
foreach (var stroke in m_dryStrokes)
{
// Draw current stroke
context.SetTarget(mRenderer.CurrentStrokeLayer);
context.ClearColor(Colors.Transparent);
DoRenderStroke(context, stroke, mRenderer.TranslationLayerPainted);
// Blend stroke to Scene Layer
context.SetTarget(mRenderer.SceneLayer);
context.DrawLayer(mRenderer.CurrentStrokeLayer, null, Ink.Rendering.BlendMode.SourceOver);
// Blend Current Stroke to All Strokes Layer
context.SetTarget(mRenderer.AllStrokesLayer);
context.DrawLayer(mRenderer.CurrentStrokeLayer, null, Ink.Rendering.BlendMode.SourceOver);
}
}
...
/// <summary>
/// Draw stroke
/// </summary>
/// <param name="renderingContext">RenderingContext to draw to</param>
/// <param name="o">Cached stroke (as object)</param>
public override void DoRenderStroke(RenderingContext renderingContext, object o, bool translationLayerPainted)
{
RasterInkStroke stroke = (RasterInkStroke)o;
renderingContext.DrawParticleStroke(stroke.Path, stroke.StrokeConstants,
ActiveTool.Brush, Ink.Rendering.BlendMode.SourceOver,
stroke.RandomSeed);
}
draw(pathPart) {
this.drawPath(pathPart);
if (pathPart.phase == InkBuilder.Phase.END) {
if (this.strokeRenderer) {
let stroke = this.strokeRenderer.toStroke(this.builder);
this.dataModel.add(stroke)
}
}
}
drawPath(pathPart) {
this.strokeRenderer.draw(pathPart.added, pathPart.phase == InkBuilder.Phase.END);
if (pathPart.phase == InkBuilder.Phase.UPDATE) {
this.strokeRenderer.drawPreliminary(pathPart.predicted);
let dirtyArea = this.canvas.bounds.intersect(this.strokeRenderer.updatedArea);
if (dirtyArea) {
this.canvas.clear(dirtyArea);
this.canvas.blend(this.strokesLayer, {rect: dirtyArea});
this.strokeRenderer.blendUpdatedArea();
}
}
else if (pathPart.phase == InkBuilder.Phase.END) {
let dirtyArea = this.canvas.bounds.intersect(this.strokeRenderer.strokeBounds.union(this.strokeRenderer.updatedArea));
if (dirtyArea) {
dirtyArea = dirtyArea.ceil();
if (!this.selector && !this.intersector) {
if (app.type == app.Type.VECTOR) {
let size = config.getSize(this.toolID);
if (size.max < 2 || size.min < 1)
this.strokeRenderer.draw(this.builder.getInkPath(), true);
}
this.strokeRenderer.blendStroke(this.strokesLayer);
this.canvas.clear(dirtyArea);
this.canvas.blend(this.strokesLayer, {rect: dirtyArea});
}
}
}
}
Vector Ink
This section describes the best practice for setting up the geometry and rendering pipeline for Vector Ink using a step-by-step approach.
Brush configuration
Vector Ink brush configuration is defined by an arbitrary convex polygon:
import com.wacom.ink.format.enums.RotationMode
import com.wacom.ink.format.rendering.RasterBrush
import com.wacom.ink.rendering.BlendMode
import com.wacom.ink.rendering.BrushPrototype
import com.wacom.ink.rendering.VectorBrush
...
class BrushPalette {
...
fun circle(): VectorBrush {
return VectorBrush(
"circle", // inSampleSize
arrayOf(
BrushPrototype(FloatArrayList(ShapeFactory.createCircle(4)), 0f),
BrushPrototype(FloatArrayList(ShapeFactory.createCircle(8)), 2f),
BrushPrototype(FloatArrayList(ShapeFactory.createCircle(16)), 6f),
BrushPrototype(FloatArrayList(ShapeFactory.createCircle(32)), 18f)
)
) // Brush prototypes for different level of details
}
...
class ShapeFactory {
companion object {
/**
* Creates polygon with circle shape
*
* @param {int} [n=20] number of polygon segments
* @param {int} [r=1] radius (0 < r <= 1)
* @param {Point} [c={x: 0, y: 0}] center
* @return {Polygon} circle shape points
*/
@JvmStatic
fun createCircle(n: Int = 20, r: Float = 1f,
c: PointF = PointF(0f, 0f)): FloatArray {
return createEllipse(n, r, r, c);
}
...
}
}
public class VectorBrushFactory
{
...
public static List<Vector2> CreateEllipseBrush(int pointsNum, float width, float height)
{
List<Vector2> brushPoints = new List<Vector2>();
double radiansStep = Math.PI * 2 / pointsNum;
double currentRadian = 0.0;
for (var i = 0; i < pointsNum; i++)
{
currentRadian = i * radiansStep;
brushPoints.Add(new Vector2((float)(width * Math.Cos(currentRadian)),
(float)(height * Math.Sin(currentRadian))));
}
return brushPoints;
}
...
}
circle: new Brush2D(URIBuilder.getBrushURI("vector", "Circle"), [
BrushPrototype.create(BrushPrototype.Type.CIRCLE, 0, 4),
BrushPrototype.create(BrushPrototype.Type.CIRCLE, 2, 8),
BrushPrototype.create(BrushPrototype.Type.CIRCLE, 6, 16),
BrushPrototype.create(BrushPrototype.Type.CIRCLE, 18, 32)
])
PathPoint calculators
The PathPoint calculators are defined in a similar way as for the raster tools. Again, we need to define the layout of the PathPoints:
override fun getLayout(): PathPointLayout {
if (isStylus) {
/**
* The currently layout will use:
* - Coordinate values (X and Y)
* - Size - the size of the brush at any point of the stroke
* - For tilt effect the rotation, scale y and offset y
*/
return PathPointLayout(
PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.SIZE,
PathPoint.Property.ROTATION,
PathPoint.Property.SCALE_Y,
PathPoint.Property.OFFSET_Y
)
} else {
/**
* The currently layout will use:
* - Coordinate values (X and Y)
* - Size - the size of the brush at any point of the stroke
*/
return PathPointLayout(
PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.SIZE
)
}
}
/// <summary>
/// Vector drawing tool for rendering pen-style output
/// </summary>
class PenTool : VectorDrawingTool
{
...
public override PathPointLayout GetLayout(Windows.Devices.Input.PointerDeviceType deviceType)
{
switch (deviceType)
{
case Windows.Devices.Input.PointerDeviceType.Mouse:
case Windows.Devices.Input.PointerDeviceType.Touch:
return new PathPointLayout(PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.Size);
case Windows.Devices.Input.PointerDeviceType.Pen:
return new PathPointLayout(PathPoint.Property.X,
PathPoint.Property.Y,
PathPoint.Property.Size,
PathPoint.Property.OffsetY);
default:
throw new Exception("Unknown input device type");
}
}
TBD
Pipeline
The VectorInkBuilder is a class that handles the path building, using the pipeline.
It processes the input to the whole pipeline in order to produce polygons from the input.
After that, the polygons are converted to native structures.
import com.wacom.ink.PathSegment
import com.wacom.ink.Phase
import com.wacom.ink.PointerData
import com.wacom.ink.Spline
import com.wacom.ink.pipeline.*
import com.wacom.ink.pipeline.base.ProcessorResult
...
class VectorInkBuilder() {
// The pipeline components:
internal lateinit var pathProducer: PathProducer
internal lateinit var smoother: SmoothingFilter
internal lateinit var splineProducer: SplineProducer
internal lateinit var splineInterpolator: SplineInterpolator
internal lateinit var brushApplier: BrushApplier
internal lateinit var convexHullChainProducer: ConvexHullChainProducer
internal lateinit var polygonMerger: PolygonMerger
internal lateinit var polygonSimplifier: PolygonSimplifier
internal lateinit var bezierPathBuilder: PolygonToBezierPathProducer
internal lateinit var pathSegment: PathSegment
lateinit var tool: VectorTool
fun updatePipeline(newTool: Tool) {
if (newTool !is VectorTool) {
throw IllegalArgumentException("Invalid tool")
} else {
tool = newTool
var layout = tool.getLayout()
pathProducer = PathProducer(layout, tool.getCalculator())
smoother = SmoothingFilter(
layout.count(), // Dimension
30) // Moving average window size
splineProducer = SplineProducer(layout) // Layout
splineInterpolator = SplineInterpolator(
tool.brush.spacing, // Spacing between two successive sample.
6, // Determines the number of iterations for the discretization.
false, // Calculate derivatives
false) // Interpolate by length
brushApplier = BrushApplier(
tool.brush, // Vector brush
true) // Keep all data
convexHullChainProducer = ConvexHullChainProducer()
polygonMerger = PolygonMerger()
polygonSimplifier = PolygonSimplifier(0.1f) // Epsilon
bezierPathBuilder = PolygonToBezierPathProducer()
pathSegment = PathSegment()
splineProducer.keepAllData = true
}
}
...
/// <summary>
/// Manages ink geometry pipeline for vector brushes.
/// </summary>
public class VectorInkBuilder : InkBuilder
{
...
#region Calculators
public void UpdatePipeline(PathPointLayout layout, Calculator calculator, VectorBrush brush)
{
bool layoutChanged = false;
if ((Layout == null) || (layout.ChannelMask != Layout.ChannelMask))
{
Layout = layout;
layoutChanged = true;
}
if (mPathProducer == null || calculator != mPathProducer.PathPointCalculator || layoutChanged)
{
mPathProducer = new PathProducer(Layout, calculator)
{
KeepAllData = true
};
}
if (mSmoothingFilter == null || layoutChanged)
{
mSmoothingFilter = new SmoothingFilter(Layout.Count)
{
KeepAllData = true
};
}
if (SplineProducer == null || layoutChanged)
{
SplineProducer = new SplineProducer(Layout)
{
KeepAllData = true
};
}
if (SplineInterpolator == null || layoutChanged)
{
SplineInterpolator = new CurvatureBasedInterpolator(Layout)
{
KeepAllData = true
};
}
if (BrushApplier == null || (brush != BrushApplier.Prototype) || layoutChanged)
{
BrushApplier = new BrushApplier(Layout, brush)
{
KeepAllData = true
};
}
if (ConvexHullChainProducer == null)
{
ConvexHullChainProducer = new ConvexHullChainProducer()
{
KeepAllData = true
};
}
if (mPolygonMerger == null)
{
mPolygonMerger = new PolygonMerger()
{
KeepAllData = true
};
}
if (PolygonSimplifier == null)
{
PolygonSimplifier = new PolygonSimplifier(0.1f)
{
KeepAllData = true
};
}
}
#endregion
// In the JavaScript implementation the setup of the pipeline within InkBuilderAbstract which is inheritat by InkBuilder
this.builder = new InkBuilder();
Vector View
Next, the vector view needs to be setup. This class is a simple view that is used to draw vector inking. Vector inking is a Path, so it is drawn by extending the platform rendering method and drawing the path there.
import com.wacom.ink.PathPoint
import com.wacom.ink.Phase
import com.wacom.ink.PointerData
import com.wacom.ink.Spline
import com.wacom.ink.format.InkSensorType
import com.wacom.ink.format.enums.InkInputType
import com.wacom.ink.manipulation.ManipulationMode
import com.wacom.ink.manipulation.SpatialModel
import com.wacom.ink.manipulation.callbacks.ErasingCallback
import com.wacom.ink.manipulation.callbacks.SelectingCallback
import com.wacom.ink.model.InkStroke
import com.wacom.ink.model.StrokeAttributes
...
class VectorView @JvmOverloads constructor(
context: Context,
attrs: AttributeSet? = null,
defStyleAttr: Int = 0
) : View(context, attrs, defStyleAttr) {
// Relevant for serialization
var inkSensorTypes = listOf(InkSensorType.X, InkSensorType.Y, InkSensorType.TIMESTAMP)
lateinit var inkEnvironmentModel: InkEnvironmentModel
var drawingTool: VectorTool = PenTool() //default tool
var inkBuilder = VectorInkBuilder()
// This will contain the current drawing path before it is finished
private var currentPath = Path()
var paint: Paint = Paint()
// A list of existing strokes
private var strokes = mutableMapOf<String, WillStroke>()
...
private fun addStroke(event: MotionEvent) {
val brush = inkBuilder.tool.brush
val stroke = WillStroke(inkBuilder.splineProducer.allData!!.copy(), brush)
stroke.sensorData = inkEnvironmentModel.createSensorData(event, inkSensorTypes)
stroke.strokeAttributes = object : StrokeAttributes {
override var size: Float = 10f
override var rotation: Float = 0.0f
override var scaleX: Float = 1.0f
override var scaleY: Float = 1.0f
override var scaleZ: Float = 1.0f
override var offsetX: Float = 0.0f
override var offsetY: Float = 0.0f
override var offsetZ: Float = 0.0f
override var red: Float = Color.red(paint.color) / 255f
override var green: Float = Color.green(paint.color) / 255f
override var blue: Float = Color.blue(paint.color) / 255f
override var alpha: Float = paint.alpha.toFloat() / 255
}
spatialModel.add(stroke)
stroke.path = currentPath
strokes[stroke.id] = stroke
}
/// <summary>
/// Creates and manages rendering brush and graphics objects and interactions between them;
/// Recieves pointer input events and passes them on to the rendering brush to feed into
/// the ink geometry pipeline
/// </summary>
public class Renderer
{
#region Fields
private SwapChainPanel m_swapChainPanel;
private PointerManager m_pointerManager;
private Graphics m_graphics = new Graphics();
#endregion
...
#region Constructor
...
/// <summary>
/// Constructor. Creates rendering brush; initializes graphics
/// </summary>
/// <param name="swapChainPanel">SwapChainPanel on which to render captured ink</param>
/// <param name="brushType">Type of brush to use</param>
/// <param name="thickness">Relative thickness of brush</param>
/// <param name="color">Color of ink</param>
/// <param name="style">Shape of brush (VectorBrush only)</param>
public Renderer(SwapChainPanel swapChainPanel, VectorBrushStyle style, MediaColor color)
{
StrokeHandler = new VectorStrokeHandler(this, style, color);
m_swapChainPanel = swapChainPanel;
m_graphics.GraphicsReady += OnGraphicsReady;
m_graphics.Initialize(m_swapChainPanel, false);
}
#endregion
class InkCanvasVector extends InkCanvas {
constructor(canvas, width, height) {
super();
this.canvas = InkCanvas2D.createInstance(canvas, width, height);
this.strokesLayer = this.canvas.createLayer();
this.strokeRenderer = new StrokeRenderer2D(this.canvas);
this.canvasTransformer = new CanvasTransformer(width, height);
this.selection = new SelectionVector(this.dataModel, {
canvas: this.canvas,
canvasTransformer: this.canvasTransformer,
redraw: this.redraw.bind(this)
});
this.selection.connect();
}
Input Handling
Input events from the operating system are handled in the same way as for the raster input (please see Raster Input Handling)
Finally, draw the stroke:
/**
* In order to make the strokes look nice and smooth, it is recommended to enable:
* - Antialiasing
* - Dither
*/
val paint = Paint().also {
it.isAntiAlias = true
it.isDither = true
}
...
override fun onDraw(canvas: Canvas) {
super.onDraw(canvas)
selectionBox?.drawSelectionBox(canvas)
for ((id, stroke) in strokes) {
if (id in selectedStrokes) {
canvas.drawPath(stroke.path, paintSelected)
} else {
// Selection color
val color = Color.argb(
(stroke.strokeAttributes.alpha * 255).toInt(),
(stroke.strokeAttributes.red * 255).toInt(),
(stroke.strokeAttributes.green * 255).toInt(),
(stroke.strokeAttributes.blue * 255).toInt()
)
paint.color = color
canvas.drawPath(stroke.path, paint)
}
}
// draw the current path,
// because we have not finished the stroke we has not saved it yet
if (drawingTool.uri() == EraserVectorTool.uri) {
paint.color = Color.WHITE
} else if (drawingTool.uri() == EraserWholeStrokeTool.uri) {
paint.color = Color.RED
} else {
paint.color = currentColor
}
canvas.drawPath(currentPath, paint)
}
public override void RenderAllStrokes(RenderingContext context,
IEnumerable<Identifier> excluded,
Rect? clipRect)
{
foreach (var stroke in m_dryStrokes)
{
if (excluded == null || !excluded.Contains(stroke.Id))
{
// Draw current stroke
context.SetTarget(mRenderer.CurrentStrokeLayer);
context.ClearColor(Colors.Transparent);
DoRenderStroke(context, stroke, mRenderer.TranslationLayerPainted);
// Blend stroke to Scene Layer
context.SetTarget(mRenderer.SceneLayer);
context.DrawLayer(mRenderer.CurrentStrokeLayer, null,
Ink.Rendering.BlendMode.SourceOver);
// Blend Current Stroke to All Strokes Layer
context.SetTarget(mRenderer.AllStrokesLayer);
context.DrawLayer(mRenderer.CurrentStrokeLayer, null,
Ink.Rendering.BlendMode.SourceOver);
}
}
}
/// <summary>
/// Handles brush-specific parts of drawing a new stroke segment
/// </summary>
/// <param name="updateRect">returns bounding rectangle of area requiring update</param>
public override void DoRenderNewStrokeSegment(out Rect updateRect)
{
var result = ActiveTool.Polygons;
ConvertPolygon(result.Addition, m_addedPolygon);
ConvertPolygon(result.Prediction, m_predictedPolygon);
// Draw the added stroke
mRenderer.RenderingContext.SetTarget(mRenderer.CurrentStrokeLayer);
Rect addedStrokeRect = mRenderer.RenderingContext.FillPolygon(m_addedPolygon, BrushColor,
Ink.Rendering.BlendMode.Max);
// Measure the predicted stroke
Rect predictedStrokeRect = mRenderer.RenderingContext.MeasurePolygonBounds(m_predictedPolygon);
// Calculate the update rect for this frame
updateRect = mRenderer.DirtyRectManager.GetUpdateRect(addedStrokeRect, predictedStrokeRect);
// Draw the predicted stroke
mRenderer.RenderingContext.SetTarget(mRenderer.PrelimPathLayer);
mRenderer.RenderingContext.DrawLayerAtPoint(mRenderer.CurrentStrokeLayer, updateRect,
new Point(updateRect.X, updateRect.Y),
Ink.Rendering.BlendMode.Copy);
mRenderer.RenderingContext.FillPolygon(m_predictedPolygon, BrushColor,
Ink.Rendering.BlendMode.Max);
}
draw(pathPart) {
this.drawPath(pathPart);
if (pathPart.phase == InkBuilder.Phase.END) {
if (this.strokeRenderer) {
let stroke = this.strokeRenderer.toStroke(this.builder);
this.dataModel.add(stroke)
}
}
}
drawPath(pathPart) {
this.strokeRenderer.draw(pathPart.added, pathPart.phase == InkBuilder.Phase.END);
if (pathPart.phase == InkBuilder.Phase.UPDATE) {
this.strokeRenderer.drawPreliminary(pathPart.predicted);
let dirtyArea = this.canvas.bounds.intersect(this.strokeRenderer.updatedArea);
if (dirtyArea) {
this.canvas.clear(dirtyArea);
this.canvas.blend(this.strokesLayer, {rect: dirtyArea});
this.strokeRenderer.blendUpdatedArea();
}
}
else if (pathPart.phase == InkBuilder.Phase.END) {
let dirtyArea = this.canvas.bounds.intersect(this.strokeRenderer.strokeBounds.union(this.strokeRenderer.updatedArea));
if (dirtyArea) {
dirtyArea = dirtyArea.ceil();
if (!this.selector && !this.intersector) {
if (app.type == app.Type.VECTOR) {
let size = config.getSize(this.toolID);
if (size.max < 2 || size.min < 1)
this.strokeRenderer.draw(this.builder.getInkPath(), true);
}
this.strokeRenderer.blendStroke(this.strokesLayer);
this.canvas.clear(dirtyArea);
this.canvas.blend(this.strokesLayer, {rect: dirtyArea});
}
}
}
}