Operator Dashboard

Every script can show a fullscreen operator dashboard. You declare its content with the global ui module. The dashboard fits the screen without scrolling. It follows the OneWare Studio theme and shows the ONE WARE branding automatically.

Widgets

Every widget has an id. If you call a widget function again with the same id, the existing widget is updated instead of adding a new one. This is how you show new results:

ui.title('Part inspection');
ui.label('verdict', 'Waiting...');

// later
ui.label('verdict', 'PASS', { color: 'lime' });

Function	Shows
ui.title(text)	The headline of the dashboard.
ui.image(id, image, options?)	An image with optional overlays. Pass null for an empty placeholder.
ui.label(id, text, options?)	A single line of text, for example the verdict. Options: color, fontSize.
ui.gauge(id, value, options?)	A value as a bar, for example a confidence. Options: min, max, color.
ui.table(id, rows, options?)	Rows of values with six rows per page.
ui.append(id, line)	Appends a line to a paginated history.
ui.list(id, items, onChange, options?)	A compact list with previous and next buttons, for walking through findings.
ui.button(id, text, onPress, options?)	A single button. Option: enabled.
ui.actions(id, buttons, options?)	Several buttons in one row.
ui.input(id, value, onChange, options?)	A text input. Options: placeholder, multiline, validate.
ui.number(id, value, onChange, options?)	A numeric input. Options: min, max, step, decimals.
ui.slider(id, value, onChange, options?)	A slider. Options: min, max, step.
ui.select(id, items, onChange, options?)	A dropdown with a searchable picker. Option: selected.
ui.checkbox(id, value, onChange, options?)	A checkbox captioned with its title.
ui.date(id, value, onChange, options?)	A date picker with yyyy-MM-dd values. Options: format, min, max.
ui.busy(id, busy)	A progress indicator over an image widget.
ui.clear()	Removes all screens and widgets.

Every widget accepts a title that is shown above it. Tables accept an array of objects, where the keys of the first row become the columns, or an array of arrays together with { columns: [...] }.

Images and overlays

ui.image shows an image in a pan-and-zoom viewer. The operator scrolls to zoom, drags to pan and clicks Fit image to reset. Overlays are drawn in image pixel coordinates:

ui.image('result', frame, {
    title: 'Inspected part',
    overlays: [
        { kind: 'rectangle', x: 40, y: 60, width: 120, height: 80, label: 'scratch', color: 'red' },
        { kind: 'point', x: 200, y: 150, radius: 6, color: 'lime' },
        { kind: 'text', x: 10, y: 20, label: 'Batch 42', fontSize: 18 }
    ]
});

Object detection results (prediction.boxes) can be passed directly as overlays. Components from the shape module can be converted with shape.toOverlays(components).

To zoom to a region, for example the finding the operator selected, pass focus:

ui.image('result', frame, { focus: { x: d.x, y: d.y, width: d.width, height: d.height } });
ui.image('result', frame, { focus: null });   // fit the whole image again

If you leave out focus, the current zoom and pan stay unchanged. This means the operator's own zooming is not reset by new frames.

tip

A widget keeps its own copy of the image. You can call image.release(frame) right after ui.image(...).

Reacting to user input

Callbacks of buttons and inputs run when the operator uses the control. Declaring or updating a widget does not call its callback. Keep callbacks short. Set a variable in the callback and do the actual work in the main loop:

let requested = false;
ui.button('inspect', 'Inspect', () => { requested = true; });

while (app.isRunning) {
    if (requested) {
        requested = false;
        ui.button('inspect', 'Inspecting...', () => {}, { enabled: false });
        await inspect();
        ui.button('inspect', 'Inspect', () => { requested = true; });
    }
    await time.sleep(30);
}

Disable a button with { enabled: false } while work is running, so the operator cannot start it twice. If you update a button, its callback is replaced as well.

Selections

ui.select and ui.list accept plain strings or objects with a value and a label. Values must be unique and must not be empty:

ui.select('reference', [
    { value: 'part-a.png', label: 'Part A' },
    { value: 'part-b.png', label: 'Part B' }
], value => { referencePath = value; }, { title: 'Reference', selected: 'part-a.png' });

If you leave out selected when you update the widget, the current selection is kept as long as it still exists.

Reviewing findings

Use ui.list instead of a table when the operator walks through findings one by one. A list keeps its natural height, so the images above it stay large:

ui.list('defects',
    defects.map((d, i) => ({
        value: String(d.id),
        label: `#${i + 1}`,
        detail: `${d.area} px`,
        color: 'red',
        muted: d.removed
    })),
    value => { selected = value; },
    { title: 'Findings', width: 'full' });

muted shows an entry struck through, for example a finding the operator rejected.

Validating input

ui.input('serial', serial, value => { serial = value; }, {
    title: 'Serial number',
    placeholder: 'SN-000000',
    validate: value => /^SN-\d{6}$/.test(value) ? null : 'Format: SN-123456'
});

Invalid values show the error next to the field and do not call the callback.

Layout

The dashboard is a grid that is filled row by row in the order you declare the widgets. The default is two columns. Change it with ui.columns(count) or per screen.

Use width to give a widget a share of the row. Each row can be divided differently:

ui.image('template', reference, { width: 'half' });
ui.image('capture', frame, { width: 'half' });
ui.label('count', '0 defects', { width: 'third' });
ui.label('area', '0 px', { width: 'third' });
ui.label('score', '0.99', { width: 'third' });
ui.actions('review', buttons, { width: 'full' });

width accepts 'full', 'three-quarters', 'two-thirds', 'half', 'third', 'quarter', 'fifth', 'sixth' or a number between 0 and 1. Every widget also accepts:

• columnSpan: the number of screen columns. width takes precedence over it.
• rowSpan: the number of rows, for example to make an image twice as high.

Images and tables take the height that is left after labels, buttons and inputs. Keep at most two rows of buttons under an image. Use ui.actions to put several buttons in one row:

ui.actions('reviewActions', [
    { text: 'Accept', onPress: accept, color: 'green' },
    { text: 'Reject', onPress: reject, color: 'red' },
    { text: 'Next part', onPress: next }
], { width: 'full' });

Screens

Split a workflow into screens, one per operator task. ui.screen declares a screen. All widgets declared afterwards belong to it. The first screen is shown automatically. ui.showScreen switches screens:

ui.title('Quality control');

ui.screen('capture', {
    title: '01 / Capture',
    description: 'Position the part and capture a frame.',
    columns: 2
});
ui.image('live', null, { title: 'Live camera', columnSpan: 2 });
ui.button('inspect', 'Capture & inspect', () => { requested = true; }, { columnSpan: 2 });

ui.screen('result', { title: '02 / Result', columns: 2 });
ui.label('verdict', 'Not inspected', { columnSpan: 2 });
ui.image('resultImage', null, { columnSpan: 2 });
ui.button('next', 'Next inspection', () => ui.showScreen('capture'), { columnSpan: 2 });

// after an inspection
ui.label('verdict', 'PASS', { color: 'lime' });
ui.showScreen('result');

Widget ids are unique across all screens. Updating live updates the widget on the capture screen, even while the result screen is shown. Screens keep their inputs, images and page state when you switch between them.

Live camera preview

Use camera.preview to show a live camera stream in an image widget. The preview runs natively and continues while your script is busy, for example during inference:

const cameras = camera.list();
log.info(cameras.map(c => `${c.id}: ${c.name}`).join(', '));

await camera.preview('live', cameras[0].id, { fps: 30, title: 'Live camera', columnSpan: 2 });

• The camera can be selected by its id, name or index. Without a camera, the first detected camera is used.
• await camera.preview(...) returns as soon as the first frame is visible. Enable inspection buttons only after it returns.
• fps limits the preview rate (1–120, default 30).
• await camera.stopPreview('live') stops the preview and keeps the last frame visible. Always stop the preview before you show a captured frame in the same widget with ui.image.

camera.capture takes a new, independent frame. It does not copy the frame that the preview shows:

await camera.stopPreview('live');
const frame = await camera.capture(cameras[0].id);
ui.image('live', frame, { title: 'Captured frame' });

Connect cameras before you start the script. Camera discovery runs once per run, and all cameras are closed when the run ends.

Recorded videos as camera

camera.video turns a video file into a looping camera. Use it to develop and demonstrate a workflow without hardware:

const replay = camera.video('./recordings/parts.mp4', { name: 'Recorded parts', fps: 30 });
await camera.preview('live', replay.id, { title: replay.name });
const frame = await camera.capture(replay.id);

A video camera advances one frame per capture. Set the preview fps at least as high as the video frame rate for real-time playback.

Showing progress

Inference and image processing block the script while they run. Show a busy indicator over the image so the operator can tell that the script is working:

ui.busy('result', 'Inspecting...');
await time.sleep(1);   // let the dashboard draw the indicator
const prediction = onnx.predict('Models/model.onnx', frame);
ui.busy('result', false);

The await time.sleep(1) is required. Without it, the dashboard has no chance to draw the indicator before the work starts.