Add AR Overlays in MatrixScan#

Prerequisites#

To proceed, you need to setup a project that uses matrix scan first, check out this guide (you can ignore the bottom section about visualization of tracked barcodes using BarcodeTrackingBasicOverlay).

Getting started#

There are two ways to add advanced AR overlays to a Data Capture View:

Note

The first way is the easiest, as it takes care of adding, removing and animating the overlay’s views whenever needed. It’s also flexible enough to cover the majority of use cases.

Using BarcodeTrackingAdvancedOverlay#

As mentioned above, the advanced overlay combined with its listener offers an easy way of adding augmentations to your DataCaptureView. In this guide we will add a view above each barcode showing its content.

First of all, create a new instance of BarcodeTrackingAdvancedOverlay and add it to the DataCaptureView.

BarcodeTrackingAdvancedOverlay overlay = BarcodeTrackingAdvancedOverlay.newInstance(barcodeTracking, dataCaptureView);

At this point, you have two options.

Note

The second way will take priority over the first one, which means that if a view for a barcode has been set using BarcodeTrackingAdvancedOverlay.setViewForTrackedBarcode(), the function BarcodeTrackingAdvancedOverlayListener.viewForTrackedBarcode() won’t be invoked for that specific barcode.

Using BarcodeTrackingAdvancedOverlayListener

@Nullable
@Override
public View viewForTrackedBarcode(
    @NotNull BarcodeTrackingAdvancedOverlay overlay,
    @NotNull TrackedBarcode trackedBarcode
) {
    // Create and return the view you want to show for this tracked barcode. You can also return null, to have no view for this barcode.
    TextView textView = new TextView(this);
    textView.setBackgroundColor(Color.WHITE);
    textView.setLayoutParams(
        new ViewGroup.LayoutParams(
            ViewGroup.LayoutParams.WRAP_CONTENT,
            ViewGroup.LayoutParams.WRAP_CONTENT
        )
    );
    textView.setText(trackedBarcode.getBarcode().getData());
    return textView;
}

@NotNull
@Override
public Anchor anchorForTrackedBarcode(
    @NotNull BarcodeTrackingAdvancedOverlay overlay,
    @NotNull TrackedBarcode trackedBarcode
) {
    // As we want the view to be above the barcode, we anchor the view's center to the top-center of the barcode quadrilateral.
    // Use the function below to adjust the position of the view by providing an offset.
    return Anchor.TOP_CENTER;
}

@NotNull
@Override
public PointWithUnit offsetForTrackedBarcode(
    @NotNull BarcodeTrackingAdvancedOverlay overlay,
    @NotNull TrackedBarcode trackedBarcode,
    @NotNull View view
) {
    // This is the offset that will be applied to the view.
    // You can use MeasureUnit.FRACTION to give a measure relative to the view itself, the sdk will take care of transforming this into pixel size.
    // We now center horizontally and move up the view to make sure it's centered and above the barcode quadrilateral by half of the view's height.
    return new PointWithUnit(
        new FloatWithUnit(0f, MeasureUnit.FRACTION),
        new FloatWithUnit(-1f, MeasureUnit.FRACTION)
    );
}

Using the setters in the overlay

The function BarcodeTrackingListener.onSessionUpdated() gives you access to a session, which contains all added, updated and removed tracked barcodes. From here you can create the view you want to display, and then call BarcodeTrackingAdvancedOverlay.setViewForTrackedBarcode(), BarcodeTrackingAdvancedOverlay.setAnchorForTrackedBarcode() and BarcodeTrackingAdvancedOverlay.setOffsetForTrackedBarcode()

@Override
public void onSessionUpdated(
    @NonNull BarcodeTracking mode,
    @NonNull final BarcodeTrackingSession session,
    @NonNull FrameData data
) {
      // Be careful, this function is not invoked on the main thread!
      runOnUiThread(new Runnable() {
          @Override
          public void run() {
              for (TrackedBarcode trackedBarcode : session.getAddedTrackedBarcodes()) {
                  TextView textView = new TextView(this);
                  textView.setBackgroundColor(Color.WHITE);
                  textView.setLayoutParams(
                      new ViewGroup.LayoutParams(
                          ViewGroup.LayoutParams.WRAP_CONTENT,
                          ViewGroup.LayoutParams.WRAP_CONTENT
                      )
                  );
                  textView.setText(trackedBarcode.getBarcode().getData());
                  overlay.setViewForTrackedBarcode(trackedBarcode, textView);
                  overlay.setAnchorForTrackedBarcode(
                      trackedBarcode, Anchor.TOP_CENTER
                  );
                  overlay.setOffsetForTrackedBarcode(
                      trackedBarcode,
                      new PointWithUnit(
                          new FloatWithUnit(0f, MeasureUnit.FRACTION),
                          new FloatWithUnit(-1f, MeasureUnit.FRACTION)
                      )
                  );
              }
          }
      });
  }

Provide your own custom implementation#

If you do not want to use our overlay, it is also possible to add augmented reality features based on the tracking identifier and quadrilateral coordinates that every tracked barcode has. Below are some pointers.

  • Set a BarcodeTrackingListener on the barcode tracking

  • In the BarcodeTrackingListener.onSessionUpdated() function fetch the added, updated and removed tracked barcodes.

  • Create and show the views for the added barcodes.

  • Remove the views for the lost barcodes.

  • Compare the updated barcodes of the current frame with the ones from the previous one, and change your visualizations accordingly. Instead of using Barcode.location which gives you the exact location of the barcode in the frame you should generally use TrackedBarcode.predictedLocation which gives you a predicted location for the barcode. Using the predicted location avoids lagging behind the camera feed.

  • There are some state transitions where animating from the previous location of a tracked barcode to its current position can end up with weird artifacts because the order of the location’s corners is not stable. If your visualization depends on the order of the corners (by for example drawing a rectangle connecting all corners), make sure that you query TrackedBarcode.shouldAnimateFromPreviousToNextState before animating. If the order of the corners were to change it is possible that an animation would flip the visualization, which is not what happened to the actual barcode. However, a visualization that just draws something in the center of the location does not depend on the the order of corners and can ignore this.

  • When animating your views take into consideration TrackedBarcode.deltaTimeToPrediction which tells you how long it will take the code to move to the predicted location. This time generally is about as long as it will take to process the next frame, giving you a new predicted location right as your previous animation is coming to an end.

Note

This frame coordinates needs to be mapped to view coordinates, using DataCaptureView.mapFrameQuadrilateralToView().

@Override
  public void onSessionUpdated(
      @NonNull BarcodeTracking mode,
      @NonNull final BarcodeTrackingSession session,
      @NonNull FrameData data
  ) {
      // Be careful, this function is not invoked on the main thread!
      runOnUiThread(new Runnable() {
          @Override
          public void run() {

              for (int lostTrackIdentifier : session.getRemovedTrackedBarcodes()) {
                  // You now know the identifier of the tracked barcode that has been lost. Usually here you would remove the views.
              }

              for (TrackedBarcode trackedBarcode : session.getAddedTrackedBarcodes()) {

                  // Fixed identifier for the tracked barcode,
                  Integer trackingIdentifier = trackedBarcode.getIdentifier();

                  // Current location of the tracked barcode.
                  Quadrilateral predictedLocation = trackedBarcode.getPredictedLocation();
                  Quadrilateral quadrilateral = dataCaptureView.mapFrameQuadrilateralToView(predictedLocation);

                  // You now know this new tracking's identifier and location. Usually here you would create and show the views.
              }

              for (Map.Entry<Integer, TrackedBarcode>  tracking : session.getTrackedBarcodes().entrySet()) {

                  // Fixed identifier for the tracked barcode,
                  Integer trackingIdentifier = tracking.getKey();

                  // Current location of the tracked barcode.
                  Quadrilateral predictedLocation = tracking.getValue().getPredictedLocation();
                  Quadrilateral quadrilateral = dataCaptureView.mapFrameQuadrilateralToView(predictedLocation);

                  // This boolean indicates if it's safe to animate the views to the current location.
                  boolean shouldAnimate = tracking.getValue().getShouldAnimateFromPreviousToNextState();

                  // You now know the updated location of the tracked barcode. Usually here you would animate the views to the retrieved location.
              }
          }
      });
  }