International Road Dynamics (IRD) has been looking at numerous ways in which to apply artificial intelligence (AI) to commercial vehicle operations (CVO). The aims are to improve the safety and efficiency of commercial vehicle fleets and freight movements. The results are a range of solutions that can be customised to meet users’ particular requirements.
The addition to CVO of AI and its subset of machine learning (ML) applications improves weigh-in- motion (WIM) systems’ accuracy. It enables automated visual checks and provides verification of inspection statuses. Vehicle matching for tracking commercial vehicles through large inspection or port facilities is another potential use.
Background
Commercial vehicle inspection stations have become highly automated in order to prevent queueing, which could back up into live traffic lanes and pose a safety risk. For maximum efficiency, weigh stations have been designed to preselect commercial vehicles that are likely to be overweight or pose other safety concerns before they reach the inspection facility.
WIM was one of the first technologies to be commonly used for pre-screening. It has since been joined by e-screening technology that can identify vehicles, by using automatic licence plate recognition (ALPR), onboard transponders or cellular applications, and electronically check their credentials and safety records. New physical screening technologies also continue to be developed, such as IRD’s in-road sensors for tyre safety screening.
AI encompasses a variety of technologies that seek to replicate tasks normally carried out by people. ML, when coupled with computer vision, offers opportunities to automate visual inspection tasks and refine pre-selection. Many weigh stations are already equipped with machine vision to provide images of commercial vehicles matched with the WIM information, or for providing images for vehicle identification using ALPR or optical character recognition of DOT numbers.
Concurrent trends
For many traffic management applications, the use of cloud computing is becoming more common. Web-based services make good sense when looking to process even very large quantities of data from remote locations. However, where in-the-moment decision-making is a requirement — such as in commercial vehicle pre-selection — having more intelligence at the edge is preferable.
IRD has therefore concentrated effort on developing the capabilities of edge devices. AI-specific processing units coupled with small, powerful industrial computers or intelligent cameras enable deep neural network and computer vision-based applications which operate in real time. These can be used at the roadside, where they can easily integrate with ITS electronics and WIM controllers like IRD’s iSinc.
Lane position and off-scale detection
Lane position has historically been a factor in WIM accuracy, as wheels positioned off the sensor or scale will result in inaccurate measurements. Off-scale sensors have been used to identify vehicles which are not in the correct position but, using video images, ML-based systems can perform the function non-intrusively. This enables the issue of driver compliance and scale avoidance to be addressed.
Dynamic load identification for calibration improvement
WIM systems have greater difficulty accurately weighing dynamic (shifting) loads such as liquids, livestock and cement. Empty flat decks are also considered to be dynamic due to the action of the unweighted trailer. It is good practice to exclude all of these from the samples used to calibrate WIM systems.
Dynamic loads can also pose problems when WIM calibration is automated, so for best results they should again be excluded. Equally, being able to identify static loads — box trailers, but also gravel-carrying hoppers, and live bottom and dump trailers — is important.
Effective calibration is important for CVO, as advances in WIM accuracy lead to better-targeted screening at inspection facilities and offer the potential for use of WIM for direct enforcement.
Direct enforcement involves the use of WIM data as the basis for overweight citations without subsequent verification of weights on static scales. Confidence in WIM accuracy is a barrier to the adoption of WIM for direct enforcement. The development of extremely accurate WIM systems would help to overcome this.
Continuously calibrated WIM (CCWIM) involves the use of data from static scales to continuously verify that acquired by nearby WIM devices.
Testing by IRD has proven the ability of the CCWIM concept to provide better calibration results over time than periodic testing using the repeated passage of vehicles of known weights. As a next step, AI can be added to improve recognition of dynamic and static load classes, and so identify those vehicles which need to be excluded for WIM calibration purposes.
To achieve this, IRD trained a deep neural network with specific annotated images and has achieved an identification confidence level of up to 99%. Images were captured from video cameras of the type already installed at inspection stations for providing overview images to weigh station operator software. Infrared illuminators enabled good performance in night-time conditions.
Vehicle classification
Both patented and open-source methods for classifying vehicles using ML techniques are now available for processing video content for applications such as traffic counting and vehicle classification. IRD offers a commercial solution of this type for traffic studies under the name iTheia and has also been working on commercial vehicle classification technology. For commercial vehicle classification, the ML model was trained to look for tyre type specifically.
For CVO applications, it is important to accurately identify the vehicle class for assessment against the correct regulations. In North America, for example, standards are very tight. They dictate that if in a day an hour’s worth of data is corrupted, then the whole of that day’s data has to be discounted; if a whole day’s data is corrupted, then that whole week’s has to be discounted.
While it is possible to classify vehicles by their WIM axle weights, there is the potential, in the case of certain trailer and axle configurations, for vehicles to be incorrectly classified. Being able to append corrected information from a real-time, image-based system can be a valuable addition to weigh station or virtual weigh station records. This can also help to confirm that any out-of-the-normal traffic patterns are indeed occurring and that these are not the results of in-ground system malfunction/poor calibration.
E-screening augmentation
Commercial vehicle tracking and matching
If one commercial vehicle passes another between the time vehicle records are created and the vehicles arrive at the inspection station, this can create confusion. For this reason, a number of methods are used to track vehicles. Weights from WIM sensors, inductive loop tracking, transponders and licence plate numbers can be used but can have limitations. For instance, in some jurisdictions the use of ALPR may be prohibited or limited to uses that do not include vehicle tracking and matching.
IRD has experimented with alternative methods of commercial vehicle tracking. ML is promising for this application even though matching images from different cameras presents some difficulties due to geometric and illumination changes. To be effective, methods must also allow for fast comparison without excessive computational requirements. This led IRD to explore matching based on physical characteristics and colour profile.
Feature matching works by creating a feature map from an image and comparing it with a later image. In CVO applications, the first image may be taken on the live lane and the later image on the inspection station approach ramp. An algorithm extracts keypoints from the images. Keypoint matching has an advantage over some other methods of matching, in that images from various camera sources may be used and the keypoints can be detected at different image resolutions despite some variance in lighting and camera positioning. Nevertheless, for best results these variances should be minimised and corrected to the greatest extent possible.
Feature matching was tested and evaluated using existing overview cameras at the Dumfries, Virginia weigh station. A sample indicated accuracies of greater than 90%. The accuracy improved as the number of vehicles compared was reduced — a typical scenario in CVO screening, where only a limited number of vehicles are moving through the inspection facility at a time.
Another method of matching is to create a colour profile for the truck tractor that can be compared with the colour profile from another image. In this method the tractor is first isolated from the rest of the image, then the colours representing the truck tractor are identified using a k-means algorithm. The algorithm essentially finds the mean for a given number of colours chosen for the image.
Currently, colour matching is not as reliable as matching based on key points, as there is variance in the colour captured by video depending on lighting conditions. However, IRD is continuing to experiment with colour matching methods, including analysing the Hue/Saturation/Value (HSV) colour space in image comparisons. Ultimately, colour matching may be a supplementary method to improve the accuracy of other vehicle matching techniques. Potentially, feature matching, colour matching and load classification could be fused to provide the best match using an algorithm.
IRD continues to explore deep learning methods to enhance the accuracy of re-identifying vehicles over a defined geographical area.
CVSA decal reading
IRD has developed and delivered to customers for use in live traffic environments a camera-based Commercial Vehicle Safety Alliance (CVSA) decal-reading application that uses ML pattern recognition to extract windshield-mounted decal parameters from video images of commercial vehicles. Two convolutional neural networks are used concurrently to recognise the windshield, detect the decal/colour, and then the digit, edge detection and classification.
CVSA decals are applied to the windshield by CVSA inspectors. Based on the information printed on the decal, the decal colour and whether one or both corners are clipped, it is possible to determine when the vehicle was last inspected and whether it is due to be inspected again. In an automated screening application, this information may be read from the moving vehicle in real time and a determination made whether to send the truck for inspection. If the truck has a decal indicating that it has been inspected within the last three months, it may be permitted to bypass inspection. If it has been more than three months, the inspection has expired and the station inspector will be shown the status as cause to bring the vehicle in for inspection. Decreasing the number of vehicles with expired inspections is directly related to improving the safety for commercial vehicles travelling on the road.
Prospects for adoption
The wide adoption of AI-based systems in CVO, or ITS more broadly, is not without its challenges. As a recent research report prepared for the US Federal Highway Administration put it, “Barriers and challenges to AI adoption are data accuracy, data privacy, rapidly changing landscape, and reluctance of agencies to adapt.”
Regulators are recommending that developers of AI-based systems ensure privacy is protected, that individuals be able to obtain an understandable explanation of decisions made about them by the AI-based system, and that there be a right to contest decisions resulting from automated decision-making.
Fortunately, commercial vehicle inspection systems have a history of using machine vision, strong regulatory frameworks, and systems for verification prior to issuing citations. There may be questions raised about AI systems that rely on black box models, as their decisions are not transparent. However, in CVO applications it is unlikely they would be used without subsequent verification — and in any case, prescreening applications are all subject to human verification when inspectors complete their work at the weigh station.
Agencies tasked with vehicle inspection have already seen the benefits of the previous generation of machine vision combined with the use of safety and credential databases, and they have addressed the privacy issues that accompany the use of such data. The potential efficiency and safety benefits of adding AI-based systems to their operations are likely to be compelling reasons for wide adoption of these and other AI-based solutions.
Sponsored content produced in association with International Road Dynamics
