Image Analysis & Inference

Scripts can run your exported ONE AI models and post-process their results with native image, mask and shape operations. All of these operations run natively and are fast enough for full-size camera frames.

Images

const frame = image.load('Dataset/Test/part_001.png');   // relative to the script folder
log.info(`${frame.width} x ${frame.height}`);

const small   = image.resize(frame, 320, 240);
const roi     = image.crop(frame, 100, 50, 400, 300);
const rotated = image.rotate(frame, 90);                 // 90, 180 or 270 degrees, clockwise
const square  = image.padSquare(frame, { color: 'white' });
const gray    = image.grayscale(frame);

image.save(roi, 'Results/roi.png');                      // PNG, missing folders are created

Every operation returns a new image. The original is not changed.

Besides these basics, the image module can draw shapes and text, filter (blur, convolution, noise), compute statistics, find templates and align images. See the API reference.

Running a model

Export your model as ONNX in ONE AI (Export) and copy it into the project, for example into the Models folder. Then run it with onnx.predict:

const MODEL = 'Models/model.onnx';
onnx.load(MODEL, { labels: ['Background', 'OK', 'Scratch'] });   // optional, fills the label names

const prediction = onnx.predict(MODEL, frame);

onnx.load is optional. onnx.predict loads the model on first use and keeps it open for the rest of the run.

To use the newest export automatically, use onnx.latest. It searches the Models folder next to the script:

const model = onnx.latest({ modelType: 'segmentation' });
if (!model) throw new Error('No segmentation model in ./Models');
const prediction = onnx.predict(model.path, frame);

Prediction results

The result depends on the model type:

prediction.type	Fields
classes (classification)	top: the best class. classes: all classes sorted by score. Each entry has labelId, label and score.
objects (object detection)	boxes: one entry per object with labelId, label, score, x, y, width and height
segmentation	mask: the class id of every pixel. width and height of the mask. confidence: the raw scores, if exported.

// Classification
if (prediction.top.label === 'OK' && prediction.top.score > 0.9) ui.label('verdict', 'PASS', { color: 'lime' });

// Object detection
ui.image('result', frame, { overlays: prediction.boxes });
ui.label('count', `${prediction.boxes.length} objects`);

// Segmentation
ui.image('result', mask.overlay(frame, prediction.mask, { opacity: 0.5 }));

Coordinates

Detection boxes and segmentation masks use the model input resolution, which can be smaller than the camera frame. mask.overlay scales the mask to the frame automatically. When you draw component boxes on the original frame, scale them by frame.width / prediction.mask.width and frame.height / prediction.mask.height. Area thresholds are measured in mask pixels.

Multi-image models

For multi-image models, pass the images as an array in the order of the image groups used in training, not in file name order:

const info = onnx.info(MODEL);
log.info(`Model expects ${info.inputImageCount} images: ${info.imageGroups.join(', ')}`);

const prediction = onnx.predict(MODEL, [reference, frame]);

onnx.info describes the model: its type, input size, image groups, inputs and outputs. If onnx.predict cannot feed a model, info.supported is false and info.reason explains why. In that case, you can still build the input tensors yourself with onnx.toTensor and run the model with onnx.run.

Masks

A mask is a label map with one class id per pixel: 0 is background and 1, 2, ... are your classes. This is the same format that ONE AI uses for segmentation annotations (_seg.png).

You get masks from a segmentation model, from a dataset, or by thresholding an image:

const predicted = prediction.mask;
const truth     = mask.load('Dataset/Train/part_001_seg.png');
const bright    = mask.fromImage(frame, { channel: 'luminance', min: 200 });

Thresholding by colour

mask.fromImage thresholds a single channel: luminance, red, green, blue, alpha, hue, saturation or value. For colour tasks, prefer the HSV channels. They separate the colour from the lighting:

const coloured = mask.fromImage(frame, { channel: 'saturation', min: 120 });   // coloured vs. grey
const red      = mask.fromImage(frame, { channel: 'hue', min: 340, max: 20 }); // wraps around 360°
const redPart  = mask.and(red, coloured);

Cleaning up masks

Raw model output contains noise. Clean it up before you count or measure:

Operation	Effect
mask.open(m, 1)	Removes small speckles. Use it before counting.
mask.close(m, 1)	Fills small holes.
mask.erode(m, r) / mask.dilate(m, r)	Shrinks or grows all regions.
mask.keepLargest(m)	Keeps only the largest region. Options: count, labelId, minArea.
mask.fillHoles(m)	Fills areas that are completely enclosed by a region.
mask.and(a, b) / mask.or(a, b) / mask.subtract(a, b)	Combines masks, even with different resolutions.
mask.binary(m, id)	Keeps one class and writes it as class 1.

const part = mask.fillHoles(mask.keepLargest(prediction.mask));
const defects = mask.keepLargest(defectMask, { count: 0, minArea: 250 });   // drop every region under 250 px

Like images, every mask operation returns a new mask.

Custom confidence thresholds

prediction.mask is thresholded at a confidence of 0.5. If your inspection needs a different operating point, export the model with the confidence output and threshold the raw scores:

const defects = mask.fromConfidence(prediction, 1, { min: 0.825 });

prediction.confidence is only available if the model was exported with a class-confidence or confidences output. Otherwise mask.fromConfidence reports that you need to export the model again.

Counting and measuring objects

The shape module finds connected regions in a mask and measures them:

const parts = shape.components(mask.open(prediction.mask, 1), {
    labelNames: { 1: 'screw', 2: 'nut' },
    minArea: 40
});

for (const p of parts) {
    log.info(`${p.label}: area ${p.area}, centre ${p.centroidX.toFixed(0)}/${p.centroidY.toFixed(0)}`);
}

ui.image('result', frame, { overlays: shape.toOverlays(parts, { color: 'lime' }) });
ui.table('perClass', shape.countByLabel(prediction.mask, { minArea: 40 }));

Each component has label, labelId, area, the bounding box (x, y, width, height), the centroid (centroidX, centroidY) and shape metrics: perimeter, circularity, aspectRatio and fillRatio. Filter components with plain JavaScript:

const round = parts.filter(p => p.circularity > 0.8 && p.fillRatio > 0.7);

Pass image: frame in the options to also get the mean colour of every component (meanRed, meanGreen, meanBlue, meanHue, meanSaturation, meanValue).

Touching objects

shape.components counts touching objects as one region. shape.instances splits them first:

const pieces = shape.instances(m, { labelId: 1, minDistance: 6, minDepth: 3, minArea: 40 });

• One object is split into several pieces: increase minDistance. If the extra pieces are thin parts of an object, also increase minDepth.
• Touching objects are still counted as one: decrease minDistance.

Outlines and geometry

To get the actual geometry of a part, use shape.contour, shape.convexHull, shape.minAreaRect or shape.quad. shape.quad returns the four corners of a part, ordered top-left, top-right, bottom-right, bottom-left. Combine it with image.perspectiveTransform and image.warp to rectify a tilted part onto a template:

const corners = shape.quad(part);
if (!corners) return;   // nothing found

const size = 512;
const m = image.perspectiveTransform(corners, [
    { x: 0, y: 0 }, { x: size - 1, y: 0 }, { x: size - 1, y: size - 1 }, { x: 0, y: size - 1 }
]);
const flat = image.warp(frame, m, size, size, { border: 'white' });

Comparing with a reference

For a golden-sample comparison, align the capture to a reference image, compensate brightness differences and measure the remaining difference:

const fit = image.register(reference, capture, { movingMask: part });
if (!fit.ok) {
    log.warn(`Alignment failed: ${fit.reason}`);
} else {
    const aligned  = image.warp(capture, fit.matrix, reference.width, reference.height, { border: 'white' });
    const levelled = image.matchBrightness(reference, aligned);
    const { mean } = image.residual(reference, levelled);
    ui.label('verdict', mean < 12 ? 'PASS' : 'FAIL', { color: mean < 12 ? 'lime' : 'red' });
}

Calibrate the threshold with known good parts. image.register uses feature matching for large displacements and falls back to an intensity-based method for parts with little texture. fit.mode tells you which method was used.

Reading a dataset

The dataset module reads a ONE AI project. It is read-only:

const ds = dataset.open('.');                 // project folder, .oneai file or bare Dataset folder
const labels = dataset.labels(ds);            // [{ id, name, color }]

for (const split of dataset.splits(ds)) {     // e.g. Train, Validation, Test
    for (const sample of dataset.samples(ds, split)) {
        const img = dataset.loadImage(sample);
        if (sample.hasMask) {
            const truth = dataset.loadMask(sample);
            // ...
            mask.release(truth);
        }
        image.release(img);
    }
}

For multi-image projects, sample.group contains the image group, for example top for part_01_top.png.

Performance

• Prefer native operations. mask.map and image.pixels call into JavaScript for every pixel. On a full-size image, this is slow and uses a lot of the memory budget (about 130 MB for one 1024×768 image with mask.map). Express the rule with mask.fromImage, mask.and, mask.subtract and similar functions instead.
• Release images and masks in loops. Everything is freed at the end of the run, but a loop over a whole dataset would otherwise keep every image in memory.
• The memory limit counts all allocations of a run, not the memory in use at one time. For stations that run permanently, set a higher limit or 0 under Settings → ONE AI → Scripting.
• Clean before you count. mask.open(m, 1) and a minArea filter remove most false regions.