How well it reads

Every build scores the reader twice: on drawings people made, which show how it does with real ink, and on generated drawings, which reach shapes and lines those drawings lack. Each score has a floor, and a change that drops a score below its floor fails the checks that run on every change.

On drawings people made#

The reader is scored on 300 drawings from Google's Didi dataset. People drew them over flowcharts that Graphviz had laid out, with no text in the shapes, so the chart under each drawing says what is there.

What the drawings look like#

Six of Didi's validation drawings, with what this reader reads in them: four read whole, and two not, with why. Each shape found is tinted, and each arrow read is shaded under the ink.

Why these drawings#

• Other people made them, for another purpose. Nobody drew them to suit this reader, so they cannot flatter it.
• The answers come with them. Each drawing keeps the chart it was drawn over, so nobody had to mark what is in it by hand, and nobody could mark it generously.
• They have no text, like a drawing here, where names are typed after drawing.
• The set is large and open. Didi has 58,655 drawings by 364 people, under a license that lets a slice live in this site's code, so anyone can score the reader again. Two older sets of hand-drawn flowcharts have 419 and 672 drawings.
• The reader never saw these people's drawings. Didi splits its drawings by who drew them. Of Didi's drawings, the reader was tuned only on its validation split, so the people who drew these 300 drew none of the drawings it was tuned on.
• They are spread through the whole set: a few drawings from each of many places in the dataset, over 246 different charts with 2 to 6 shapes each, laid out in all four directions.

How it is scored#

• A shape is found when a shape read from the drawing has its middle inside the rectangle the chart's shape fits in. People drew near the chart's shapes but rarely at exactly their size, so the middle is the fair test. Each shape read counts for one chart shape only, nearest first, so one big outline cannot count for two.
• A look is scored only for the chart's boxes, ovals, diamonds, and parallelograms, the shapes the reader has a look for. The charts also use octagons, which have no look of their own here. They count as found when read as any shape, and you can give them the look you want under Names.
• An arrow is right when it joins the right two shapes, pointing the right way, one way only. A line read with no head points the way it was drawn, as it does on this page. The charts never have a head at both ends, so a two-way reading is wrong.
• A line's style is scored on the arrows read right. Every line in the charts is plain, so one read as dotted or thick is a mistake, as when a person drew a line in two pieces with a gap. Arrows read as dotted or thick still count as right arrows.
• A whole drawing is right only when every shape is found with nothing extra, and every arrow is right with nothing extra. This is the figure closest to using it: nothing to add or remove, though a shape's look or the diagram's direction may still need a change.

What it cannot tell you#

• Chance moves it. From 300 drawings, the reader's whole-drawing rate on drawings like these is known only to within about 5.3 points either way (half a 95% interval). The other figures count more items, so they are known more closely. A slice this size keeps the file under 1 MB and the scoring under a second, so every build runs it.
• Copying is tidier than sketching. People drew over a chart they could see, so their shapes sit where the chart's are, at its sizes and spacing. A sketch made from your head is looser, so expect somewhat less on your own drawings.
• The charts leave things out: circles, groups, loops on one shape, arrows with a head at both ends, lines with no head, and dotted and thick lines. So these drawings show that plain lines stay plain, but not how well people's dashes and doubled lines are read. The generated drawings below have circles, lines with no head, and dotted and thick lines, and the reader's tests check groups, loops, and two heads on drawings made for each case.
• The devices differ. The drawings were made in an app on Chromebooks, with and without a stylus, and on Android devices, not on this page.

How people draw arrows#

People draw arrows in more ways than a line with a hook. Four ways came up often enough in Didi's validation drawings to need a rule of their own. Below is one drawing of each from that split, with the ink as the person drew it (rescaled, as all the drawings here are), each shape the reader found tinted, and the arrow it read shaded under the ink.

The table counts the arrows read right with every rule, without each rule in turn, and without all four, measured once with version 0.4.1. The rules were worked out on 300 validation drawings, by looking at the arrows missed there, so they gain the most on those. The 109 more were taken later from the same split and never looked at one by one, though some may be by the same people; the 300 drawings scored above are by other people. Those two columns show how much carries over.

The rules overlap, so the drops in the four rows need not add up to the drop in the last: 50 against 44 on the drawings scored above, 123 against 114 on the 300 validation drawings, and 35 against 35 on the 109 more. Closed heads matter most on the drawings scored above, and they also keep a head from reading as a shape with an arrow of its own: without them, those drawings have 95 wrong extra arrows instead of 76. The scores further up are measured again on every build; these are not.

On generated drawings#

Every build also scores the reader on 1,000 generated wobbly flowcharts with known answers: 3 to 5 shapes each, in boxes, wide ovals, circles, circles in circles, diamonds, hexagons, and trapezoids, with hooks, separate "V" heads, and lines with no head, and loops that run back up the side. About a quarter of the lines are drawn as dashes, 10 to 20 units long with gaps of 6 to 12, stretched or shrunk so a whole number fit, at least two; about a quarter are gone over again, a few units to one side, from either end. They come from the same seed every time, so two builds see the same drawings, and a change in the score comes from the reader, not from the drawings.

The same drawings are scored again with every line plain, so that reading dashes and doubled lines cannot hide a loss on plain ones: 99.8% of arrows and 98.5% of whole drawings read right, held to floors of 99.5% and 97.5%.

Every build also scores the reader on 1,000 generated state diagrams, read as state diagrams: a start, 2 to 4 states, and an end down a column, sometimes with a choice that sends a branch to one side, a group around two states, or an arrow that loops back up the side. The start is a filled dot, a small circle, a small oval, or a dot filled in at the end of its arrow; a group sometimes has a start of its own inside it; the end is a circle with a dot or a circle in it. No public set of hand-drawn state diagrams is known, so these are all the measure there is.

Generated drawings are tidier than a person's, so these figures show the reader's rules hold on clean cases, not how it does on your sketches. That is what the drawings people made are for, and no public set of drawings has people's dotted or thick lines to score. A box in a box, groups, colored ink, and the cases above that neither set has are checked by the reader's tests, each on a drawing made for it.

The Didi dataset#

The Didi dataset is by Google LLC, used under the CC BY 4.0 license, and described in Philippe Gervais, Thomas Deselaers, Emre Aksan, and Otmar Hilliges, "The DIDI dataset: Digital Ink Diagram data" (2020), where the counts and the older sets above come from. mermaidify keeps 300 of its drawings, rescaled and thinned, to test against. The drawings under What the drawings look like and How people draw arrows are from its validation split, rescaled and thinned the same way.