Introduction
AI Builder is a Power Platform capability that provides you the AI models to take leverage of artificial intelligence to optimize your business as well as automate processes. We can either use the already existing prebuilt models that suit our needs to use the Power of AI in Power Apps or Power Automate or create our own custom AI models to suit some specific business needs. You can read more about AI Models in the Official documentation here. The out of the box pre-built available models in AI Builder are:
- Business card reader
- Category classification
- Entity extraction
- ID reader
- Invoice Processing
- Key phrase extraction
- Language detection
- Receipt processing
- Sentiment analysis
- Text recognition
- Text translation
In this article, we will see how to use the AI Builder Model – Text recognition to extract text from an image in unison with a Power App.
Scenario
We will try to create a Power App that can be used on the fields for tracking speeding cars so that we can use the app to capture the Registration Number of the vehicle. We will then use the Cognitive Analysis ability of AI Builder to extract the number from the image and store it in the back end for identifying the offender and sending them notifications.
Implementation
As the front end, we will create a Power App to capture the speeding image and any other needed information regarding the speeding offense. In the middleware, we will use Power Automate and AI Builder to extract information from the captured speeding image. We will then store all the information back to a SharePoint back-end list.
Create the SharePoint List
We will create a SharePoint list with the below minimal columns to save the information
Create the Power Automate
Let’s create a flow that will accept the captured speeding image and extract the text from it. We will create the flow with the Power Apps trigger and add the Recognize text in an image AI Builder action to the flow.
Click on Ask in PowerApps which will create a variable by the name “RecognizetextinanimageorPDFdocument_Image” which can be used to pass the image from Power App to Flow. The image variable passed from the Power App which contains the base 64 encoded image will be used as the input for the AI builder action which will process it and output the results body which contains the extracted text.
The output of the AI Builder action returns the result in the below structure where the Extracted Text is embedded inside the “Lines” array
So as to extract the image text, we will parse the body using the below expression which will get the extracted text and assign it to the variable named SpeedingNumber.
| outputs(‘Recognize_text_in_an_image_or_a_PDF_document’)?[‘body/responsev2/predictionOutput/results’][0]?[‘lines’][0]?[‘text’] |
Finally, we will add a Respond to a Power App action to send back the extracted Speeding Number
Build the Power App
In the Power App, we will add the Camera Control, which on clicking will take the snap of the speeding vehicle. The captured snapshot is shown on the screen using an image control. Followed by that, we have a text area that shows the extracted speeding vehicles number which we get back from the previously created flow. Finally, we also have a field to capture the current time.
Capturing the Image
We will set the Image property of the Image Control to the Photo attribute of the Camera control so that on click of the camera, the image is shown in the image control.
In the OnSelect of the Camera Control,we will Run the Flow using <FlowName>.Run function and pass the Snapped image as the parameter to the flow which will be accepted by the variable that we had registered inside the flow. Since the flow outputs the Speeding number, we will get that and assign it to a variable using the Set function.
Set(SpeedingNumber,PowerAppSpeedingCamera.Run(Camera_SpeedingCamera.Photo).speedingnumber)
Finally, the extracted image text in the variable is assigned to the Text property of the text box to display on the screen.
We have already made a data connection to the SharePoint List which we can see in the left pane. Let’s save the offense data to the SharePoint list using the Patch function
| Patch(‘Speeding Tickets’,{‘Vehicle Number’:Label_SpeedingNumber.Text,’Offence Date’:Label_OffenceDate.Text,’Speeding Snapshot’:Camera_SpeedingCamera.Photo}) |
This will save the data to the back end and for time being we are saving the Image as base64 text in the SharePoint List. We can use Power Automate to convert it into an image and store it as well.
Testing the App
Now let’s test the app by taking a snap of the number plate of a car and see how the text is extracted from the Image and saved back to SharePoint List.
Summary
Thus we saw how to create a Speeding Ticket Power App that takes snaps of the speeding vehicles’ number plate and uses AI Builder’s cognitive ability to extract text from the image and save the offense to the SharePoint List.
