Prof. CHENG Chun Hung, Department of Systems Engineering and Engineering Management

This project analyses video streams from surveillance cameras to automatically estimate baggage trolley availability at various trolley racks or pickup points at  Hong Kong International Airport. Once the research application has proven its practicality in terms of accuracy and ease of use, such lines of research will open up numerous opportunities for automating different real-time tracking and monitoring tasks for better resource and service management, thus enhancing timely and on-the-spot workforce allocations in selected industrial and business labor-intensive operations.

Turning Videos into Tracking Data & Actionable Intelligence
This project strives to analyse the surveillance video streams of baggage trolley racks and deduce the quantities of available trolleys in a real-time manner. By feeding back trolley numbers and alerting the authorities when quantities fall below a threshold, we strive to achieve uninterrupted trolley availability for passengers at the extremely busy Hong Kong International Airport (HKIA).  Figure 1 shows the various aspects of the project: from monitoring to making estimation of the number of available trolleys and finally providing real-time estimates.

Figure 1. The enabling technologies of turning visual content from video streams into individually detected baggage trolleys in Baggage Reclaim Hall trolley racks at Hong Kong International Airport support real-time trolley availability monitoring via mobile apps for timely trolley replenishment. In addition to strengthening on-site passenger services, the cumulated efforts provide a means for resource analytics for managing and planning workforce, equipment circulation, management practices, and so on.

By using machine learning techniques such as support vector machines and neural networks, we are able to achieve a trolley detection rate of about 91% from video streams. The machine learning process involves annotating positive and negative visual samples on images extracted from video streams, representing visual samples by using selected image feature sets, iteratively going through learning cycles, and evaluating detection accuracy. Once a trained trolley detector is obtained, we then turn these imperfect detection results into reasonably reliable and stable availability quantities to serve certain dynamical operations. To achieve the goal, a variety of techniques from systems engineering and science are involved to establish innovative system architecture and design.

Collaboration with Hong Kong International Airport
This project is done in close collaboration with Hong Kong International Airport. HKIA has the service commitment to ensuring that baggage trolleys meet incoming and outgoing passengers' real-time demand. In order to keep the commitment, HKIA looks for a cost-effective system to closely monitor real-time trolley usages at all racks. Such a system should enable HKIA to provide timely trolley replenishment for individual racks having shortages. Being the pilot site for testing R&D results, HKIA plays the role of application user of the evolving trolley monitoring system. 

Deploying Proof-of-Concept Trolley Monitoring Prototype Systems
The work started with a small-scale proof-of-concept monitoring prototype system with existing non-dedicated surveillance CCTV cameras in Baggage Reclaim Hall to monitor 5 trolley racks in late 2014. After experiencing the real-time trolley availability monitoring capability with red (empty rack) or yellow (having shortage) alert notifications from an Android Trolleys monitoring app, HKIA decided to install a set of 4 dedicated IP cameras which were set up on carousel 10 for achieving full-time testing and evaluation of the image-based trolley monitoring system in the second quarter of 2015. After going through planning and installation, the dedicated camera setting went live with both iOS and Android Trolleys apps in November 2015. Figure 2 depicts the building and engineering works of installing cameras and network infrastructure in Baggage Reclaim Hall at Hong Kong International Airport. The installation took place in the early hours when the least number of passengers would be inconvenienced. With scalability built in the Trolley Monitoring System, detection results from both dedicated and non-dedicated cameras as well as image processing architectures can be incorporated together to provide timely alert notifications.

Figure 2. The installation of camera network infrastructure involves various building, cabling, and engineering works conducted by HKIA management and technical personnel, contracted engineering staff, and CUHK research team in the early hours when disruption to passenger services is at the minimum.

Running Full-scale Prototype Deployment in HKIA Baggage Reclaim Hall
Built on pilot implementations on a smaller scale, the plan for a full-scale deployment covering all 18 trolley racks in Baggage Reclaim Hall started in December 2015. The construction work for video network infrastructure in Baggage Reclaim Hall was finished at the end of October 2016. The trolley availability monitoring at all 18 racks started to operate at the beginning of November 2016.  In addition to having timely alerts for replenishing corresponding racks, the Trolley Monitoring System enables a real-time bird's eye view of trolley usages with estimated trolley quantities for individual racks as well as the entire Baggage Reclaim Hall as shown in Figure 3. This full-scale deployment creates a brand new experience of  real-time  trolley usages and availability simultaneously for both on-site operation managers and administrative management via the Trolleys apps on their mobile phones. Figure 4 illustrates three of the Trolleys app screens for viewing alert notifications, images of trolley racks, and changes of trolley quantities in a chart view.

	Figure 3. A bird's eye view or floor plan view of trolley quantities and alert statuses at each trolley rack and in the entire Baggage Reclaim Hall provides a means for operation managers to direct limited resources – workforce, trolleys, and time – to make timely replenishment to the most needed spots.
	Figure 4. Three of the functionalities in the Trolleys mobile app are illustrated: Bird's eye view showing trolley quantities available at each rack and in the entire Baggage Reclaim Hall, masked rack images for viewing the situation at a certain spot, and chart view of trolley quantities accumulated over a specific period of time.

On-going Research & Development
In addition to improving detection accuracy for large trolleys, the project continues to create appropriate trolley detectors through machine learning techniques to simultaneously deal with varying lighting conditions in indoor, outdoor, and glass-room environments as well as other types of trolley deployed at HKIA. In terms of deployment flexibility, HKIA keeps on exploring cost-effective and scalable system architecture and design to modularize trolley detection units over large multi-building facilities.

Working Towards Commercialization for Appropriate Business Applications
This applied research project not only advances the science and technology for understanding image content, but also promotes the applicability of image-based technologies for various business operations via the live and operable Trolley Monitoring System at HKIA. In fact, the image-based technologies developed in this project are applicable vertically to airports around the world, or horizontally to other CCTV-ready facilities like shopping malls and exhibition centers with indoor, outdoor, and mixed environments to monitor the allocation of resources within premises.

Several uniqueness and competitive advantages of the underlying techniques are:
1. Leveraging the existing CCTV infrastructure allows continuous monitoring and management at low installation and maintenance cost.
2. Tracking of homogeneous items can be achieved without physically tagging them with other identification technologies.
3. The bottlenecks in detection and prediction capability in the existing video content analytic techniques are avoided so that a smooth flow can be ensured throughout large, multi-stage transit centers.
4. Multimodal monitoring capability is provided for managing multi-dimensional dynamics in transit centers.
5. Various monitoring mechanisms are established for providing flexibility in collecting current statuses at different spots.
6. Prediction of chain reactions due to inefficiency cumulating in one or multiple spots will be provided.

Acknowledgements
This project was funded by Innovation and Technology Fund (ITF) and Hong Kong R&D Centre for Logistics and Supply Chain Management Enabling Technologies (LSCM). The seeded research and investigation work was funded and supported by the Asian Institute of Supply Chains & Logistics, The Chinese University of Hong Kong. The domain knowledge of passenger services and trolley replenishment operations as well as the camera network infrastructure work was supported by Hong Kong International Airport. The research and development work was carried out by the Service Science & Innovation Group directed by Prof. Chun Hung Cheng in the Department of Systems Engineering and Engineering Management at CUHK. The promotional and commercialization efforts were organized by the Centre for Innovation and Technology at CUHK.

More Information
Project website at SEEM:
• www.se.cuhk.edu.hk/~ssi/Research/Video_Analytics_for_Resource_Management
Online Exhibitions at CINTEC:
• http://exhibition.cintec.cuhk.edu.hk/exhibition/project-item/video-analytics-for-resource-management/
• http://exhibition.cintec.cuhk.edu.hk/exhibition/project-item/video-analytic-resource-management/
Project information at LSCM & ITF
• http://www.itf.gov.hk/l-eng/prj_Profile.asp?code=1B3EDA04EE1148B3E7D1B887D0D8E9C1ACAC4EB649F2119EC931C4ED2155D517
• www.lscm.hk/eng/project.php?detail_id=98