Scan multiple barcodes at once with MatrixScan

Motivation

Sometimes you have packages/pallets with several codes, with the Scandit Barcode Scanner you can scan all the codes or just a subset at the same time, saving time and money. To do this you have to use the Matrix Scan option.

Matrix Scan

Matrix Scan track recognized barcodes over time by making it easier to use.

Enabling Matrix Scan

To implement Matrix Scanning

One of the use cases of Matrix Scan is to detect when the specified number of expected codes has been decoded by the scanner. To implement this scenario:

  • In the barcodePicker:didScan: (SBSScanDelegate-p) method, wait until the number of expected codes have been decoded, then store the codes and pause/stop the session. If you pause and plan to resume but want to start a new session, make sure that you clear the session first (see code snippet below).

At this point you can start the barcode picker and any recognized barcodes of the enabled symbologies will be highlighted by a filled green rectangle. Barcodes that have been localized but not recognized will be highlighted by a green border.

Objective-C:

- (void)setupScanner {
SBSScanSettings *scanSettings = [SBSScanSettings defaultSettings];
// Enable the relevant symbologies.
[scanSettings setSymbology:SBSSymbologyEAN13 enabled:YES];
[scanSettings setSymbology:SBSSymbologyCode39 enabled:YES];
// Enable Matrix Scanning.
scanSettings.matrixScanEnabled = YES;
// The maximum number of codes to be decoded every frame.
scanSettings.maxNumberOfCodesPerFrame = 3;
// Create the picker, set the listener, add the picker to the view hierarchy and start it.
SBSBarcodePicker *picker = [[SBSBarcodePicker alloc] initWithSettings:scanSettings];
picker.scanDelegate = self;
// Set the GUI style accordingly.
picker.overlayController.guiStyle = SBSGuiStyleMatrixScan;
[self addChildViewController:picker];
picker.view.frame = self.view.bounds;
[self.view addSubview:picker.view];
[picker didMoveToParentViewController:self];
[picker startScanning];
}
- (void)barcodePicker:(SBSBarcodePicker *)picker didScan:(SBSScanSession *)session {
// Number of expected barcodes.
int numExpectedCodes = 3;
// Get all the scanned barcodes from the session.
NSArray *allCodes = session.allRecognizedCodes;
// If the number of scanned codes is greater than or equal to the number of expected barcodes,
// pause the scanning and clear the session (to remove recognized barcodes).
if ([allCodes count] >= numExpectedCodes) {
// Stop scanning or pause and clear the session.
[session stopScanning];
// ...
}
}

Swift:

func setupScanner() {
let scanSettings = SBSScanSettings.default()
// Enable the relevant symbologies.
scanSettings.setSymbology(.ean13, enabled: true)
scanSettings.setSymbology(.code39, enabled: true)
// Enable Matrix Scanning.
scanSettings.isMatrixScanEnabled = true
// The maximum number of codes to be decoded every frame.
scanSettings.maxNumberOfCodesPerFrame = 3
// Create the picker, set the listener, add the picker to the view hierarchy and start it.
let picker = SBSBarcodePicker(settings: scanSettings)
picker.scanDelegate = self
// Set the GUI style accordingly.
picker.overlayController.guiStyle = .matrixScan;
addChildViewController(picker)
picker.view.frame = view.bounds
view.addSubview(picker.view)
picker.didMove(toParentViewController: self)
picker.startScanning()
}
func barcodePicker(_ picker: SBSBarcodePicker, didScan session: SBSScanSession) {
// Number of expected barcodes.
let numExpectedCodes = 3
// Get all the scanned barcodes from the session.
let allCodes = session.allRecognizedCodes
// If the number of scanned codes is greater than or equal to the number of expected barcodes,
// pause the scanning and clear the session (to remove recognized barcodes).
if allCodes.count >= numExpectedCodes {
// Stop scanning or pause and clear the session.
session.stopScanning()
// ...
}
}

Rejecting unwanted codes

Like normal scanning Matrix Scanning provides you the option to reject codes. Just like for normal scanning rejected codes are not added to the scan session and do not provide any feedback like vibrating or beeping. In Matrix Scan the rejected codes are still drawn on the screen but in a different color. This gives the user visual feedback that a certain code is not the one you are looking for. To implement rejection

Note: Rejecting in the barcodePicker:didScan: (SBSScanDelegate-p) method is not allowed.

Objective-C:

- (void)setupScanner {
// Same initialization as above.
// Set the process frame listener.
picker.processFrameDelegate = self;
}
- (void)barcodePicker:(SBSBarcodePicker *)aPicker didProcessFrame:(CMSampleBufferRef)frame session:(SBSScanSession *)session {
for (NSNumber *identifier in session.trackedCodes) {
SBSTrackedCode *trackedCode = session.trackedCodes[identifier];
// Reject all Code39 barcodes.
if (trackedCode.symbology == SBSSymbologyCode39) {
[session rejectTrackedCode:trackedCode];
}
}
}

Swift:

func setupScanner() {
// Same initialization as above.
// Set the process frame listener.
picker.processFrameDelegate = self
}
func barcodePicker(_ barcodePicker: SBSBarcodePicker, didProcessFrame frame: CMSampleBuffer, session: SBSScanSession) {
guard let trackedCodes = session.trackedCodes else { return }
for (_, trackedCode) in trackedCodes {
if trackedCode.symbology == .code39 {
session.rejectTrackedCode(trackedCode)
}
}
}

Providing your own visualization

MatrixScan returns the locations of tracked barcodes with every processed frame. This allows you to replace the default visualization with your own. For example, you might want to overlay the barcodes with images of the products they refer to. To add your own visualization, you have two options:

  • Take advantage of the Advanced Overlays Framework, which is the high-level API for the MatrixScan that was designed and developed specifically for this purpose.
  • Implement your own mechanism for drawing and animating the visualizations for every tracked barcode.

Both of these approaches are described in the sections below. Note: The Advanced Overlays Framework is the easier and preferred method of implementing custom visualizations, as it takes care of all the animations for you and should cover the majority of use cases.

With the Advanced Overlays Framework

As mentioned before, the Advanced Overlays Framework (AOF) is a high-level interface designed to simplify the process of applying your own visualizations to MatrixScan. This framework consists of several classes and protocols that can be imported with the <ScanditBarcodeScanner/SBSMatrixScan.h> header.

Below there are some implementation guidelines that will help you better understand how the AOF works and how it should be used.

The main component of the AOF is the SBSMatrixScanHandler class. In order to use the framework you will have to instantiate it. It requires an instance of SBSBarcodePicker and an instance of SBSMatrixScanDelegate (more on that later). To add a custom overlay to be drawn on top of the tracked barcodes, simply call the addOverlay: (SBSMatrixScanHandler) method. The SBSMatrixScanOverlay can only be conformed by UIView subclasses.

To make things easier, the AOF provides two concrete implementations of the SBSMatrixScanOverlay: SBSSimpleMatrixScanOverlay and SBSViewBasedMatrixScanOverlay. The first one enables you to draw colorful, rectangle overlays on top of the tracked barcodes - similar to the way MatrixScan without AOF Framework works, except with SBSSimpleMatrixScanOverlay you get more control over the color of every tracked barcode. The SBSViewBasedMatrixScanOverlay, on the other hand, uses UIView class instances as visualizations for all of the tracked barcodes.

The AOF Framework doesn't limit you to just one custom overlay - you can call the addOverlay: (SBSMatrixScanHandler) method multiple times. You just have to remember, that the overlays will be displayed on top of each other in the order of their addition. Note: Adding too many overlays can decrease the performance and responsiveness of your app, especially on low-end devices.

Finally, you can manipulate the behavior of the AOF Framework with the SBSMatrixScanDelegate, mentioned earlier in this section. This protocol consists of 2 delegate methods. The matrixScanHandler:didProcessFrame: (SBSMatrixScanDelegate-p) method will be called every time a SBSFrame is being processed by the SBSMatrixScanHandler. The matrixScanHandler:shouldRejectCode: (SBSMatrixScanDelegate-p) method, on the other hand, can be used to define an extra condition for rejecting a barcode (this is analogous to the code rejection described in the Rejecting unwanted codes section).

SBSSimpleMatrixScanOverlay

Swift:

func simpleMatrixScanOverlay(_ overlay: SBSMatrixScanOverlay, colorFor code: SBSTrackedCode, withIdentifier identifier: NSNumber) -> UIColor {
if code.symbology == .ean13 {
// every EAN13 barcode will have a yellow augmentation drawn on top of it
return UIColor.yellow
} else {
return UIColor.green
}
}

The SBSSimpleMatrixScanOverlayDelegate also provides you with a simpleMatrixScanOverlay:didTapCode:withIdentifier: (SBSSimpleMatrixScanOverlayDelegate-p) method, that will be called every time the given augmentation is touched, so that you can perform some extra action at that time e.g.:

Swift:

func simpleMatrixScanOverlay(_ overlay: SBSSimpleMatrixScanOverlay, didTap code: SBSTrackedCode, withIdentifier identifier: NSNumber) {
let alert = UIAlertController(title: "Augmentation tapped", message: nil, preferredStyle: .alert)
alert.addAction(UIAlertAction(title: "Ok", style: .cancel) { [weak self] (_) in
self?.dismiss(animated: true, completion: nil)
})
present(alert, animated: true, completion: nil)
}

SBSViewBasedMatrixScanOverlay

Similarly to the SBSSimpleMatrixScanOverlay, the SBSViewBasedMatrixScanOverlay is also equipped with a listener: SBSViewBasedMatrixScanOverlayDelegate. It provides two useful callbacks for manipulating the view augmentation of each barcode: SBSViewBasedMatrixScanOverlay::viewBasedMatrixScanOverlay:viewForCode:withIdentifier: and SBSViewBasedMatrixScanOverlay::viewBasedMatrixScanOverlay:offsetForCode:withIdentifier:. Example: Swift:

func viewBasedMatrixScanOverlay(_ overlay: SBSViewBasedMatrixScanOverlay, viewFor code: SBSTrackedCode, withIdentifier identifier: NSNumber) -> UIView {
let view = UIView(frame: CGRect(x: 0, y: 0, width: 40, height: 80))
view.backgroundColor = .red
return view
}
func viewBasedMatrixScanOverlay(_ overlay: SBSViewBasedMatrixScanOverlay, offsetFor code: SBSTrackedCode, withIdentifier identifier: NSNumber) -> UIOffset {
// This method is called every time a new barcode has been tracked.
// You can implement this method to return the offset that will be used to position the augmentation
// with respect to the center of the tracked barcode.
if code.symbology == .ean13 {
return UIOffset.zero
} else {
return UIOffset(horizontal: 10, vertical: 20)
}
}

Note: In the documentation a visualization for a single barcode is also often called an augmentation. Therefore, a SBSViewBasedMatrixScanOverlay consists of multiple augmentations (one augmentation for every tracked barcode currently within the camera feed).

Without the Advanced Overlays Framework

If for some reason the Advanced Overlays Framework doesn't suit your needs, you can still replace the default visualization with your own, by following the steps described below

  • Set a SBSProcessFrameDelegate on the picker SBSBarcodePicker::processFrameDelegate.
  • In the barcodePicker:didProcessFrame:session: (SBSProcessFrameDelegate-p) method you fetch the currently tracked barcodes from the session through rejectTrackedCode: (SBSScanSession). The object is a dictionary with keys of type NSNumber which identify the SBSTrackedCodes over different frames.
  • Compare the keys of the current frame with the one from the previous one and change your visualization accordingly. Instead of using SBSTrackedCode::location which gives you the exact location of the barcode in the frame, you should generally use SBSTrackedCode::predictedLocation which gives you a predicted location for the barcode. Using the predicted location avoids lagging behind the camera feed that is displayed.
  • There are some state transitions where animating from the previous location of a tracked barcode to its current position can end up with weird artifacts as the edge ordering of the location is not stable. If your visualization depends on the edge ordering make sure that you query SBSTrackedCode::shouldAnimateFromPreviousToNextState before animating. The default visualization makes use of this as it draws the location with all four edges and therefore depends on the edge ordering, if the order of the edges were to change it is possible that an animation would flip the visualization which is not what happens to the actual barcode. However, a visualization that just draws something in the center of the location does not depend on the edge ordering and can ignore SBSTrackedCode::shouldAnimateFromPreviousToNextState.
  • When animating your visualization take into consideration SBSTrackedCode::deltaTimeForPrediction which tells you how long it will take the code to move to the predicted location. This time generally is about as long as it will take the barcode picker to process the next frame, giving you a new predicted location right as your previous animation is coming to an end.

Important: The coordinates you get back from SBSTrackedCode::location and SBSTrackedCode::predictedLocation are in the coordinate system of the processed frame. To visualize them on top of the picker's camera feed you have to convert them to the picker's coordinate system which you do by calling SBSBarcodePicker::convertPointToPickerCoordinates.

Important: Don't forget that the barcodePicker:didProcessFrame:session: (SBSProcessFrameDelegate-p) method is not called on the mainThread, you have to switch to it to change the UI.

Objective-C:

- (instancetype)init {
self = [super init];
if (self) {
// visualizations is a property of type NSMutableDictionary<NSNumber *, CAShapeLayer *> *
_visualizations = [NSMutableDictionary dictionary];
}
}
- (void)barcodePicker:(SBSBarcodePicker *)aPicker didProcessFrame:(CMSampleBufferRef)frame session:(SBSScanSession *)session {
// Iterate over the current visualizations and remove all that are no longer tracked.
NSDictionary<NSNumber *, CAShapeLayer *> *currentVisualizations = [self.visualizations copy];
for (NSNumber *identifier in currentVisualizations) {
if (session.trackedCodes[identifier] == nil) {
self.visualizations[identifier] = nil;
// Maybe add an animation for barcodes that vanish.
}
}
// Iterate over all the currently tracked barcodes, update old ones and add new ones.
for (NSNumber *identifier in session.trackedCodes) {
if (self.visualizations[identifier] != nil) {
// Update the visualization
SBSTrackedCode *trackedCode = session.trackedCodes[identifier];
if (trackedCode.shouldAnimateFromPreviousToNextState) {
// Animate to the new predicted location.
}
} else {
// Add a new visualization, maybe animate the appearance.
}
}
}

Swift:

// Dictionary accessed from the session thread
var visualizations: [NSNumber: CAShapeLayer] = [:]
func barcodePicker(_ barcodePicker: SBSBarcodePicker, didProcessFrame frame: CMSampleBuffer, session: SBSScanSession) {
guard let trackedCodes = session.trackedCodes else { return }
// Iterate over the current visualizations and remove all that are no longer tracked.
let currentVisualizations = visualizations
for (identifier, _) in currentVisualizations {
if trackedCodes[identifier] == nil {
self.visualizations[identifier] = nil
// Maybe add an animation for barcodes that vanish.
}
}
// Iterate over all the currently tracked barcodes, update old ones and add new ones.
for (identifier, trackedCode) in trackedCodes {
if self.visualizations[identifier] != nil {
// Update the visualization
if trackedCode.shouldAnimateFromPreviousToNextState {
// Animate to the new predicted location.
}
} else {
// Add a new visualization, maybe animate the appearance.
}
}
}
Copyright (C) 2021 Scandit AG