• Detecting threat behaviours, L. Patino and J. Ferryman. In Advanced Video and Signal Based Surveillance (AVSS), 13th IEEE International Conference on, pages 1--6. IEEE, 2016

    This paper addresses the complex problem of recognising threat situations from videos streamed by surveillance cameras. A behaviour recognition approach is proposed, which is based on a semantic recognition of the event. Low-level tracking information is transformed into high-level semantic descriptions mainly by analysis of the tracked object speed and direction. Semantic terms combined with automatically learned activity zones of the observed scene allow delivering behaviour events indicating the mobile activity. Behaviours of interest are modelled and recognised in the semantic domain. The approach has been applied on different public datasets, namely CAVIAR and ARENA. Both datasets contain instances of people attacked (with physical aggression). Successful results have been obtained when compared to other state of the art algorithms.
  • Meeting detection in video through semantic analysis, L. Patino and J. Ferryman. In Advanced Video and Signal Based Surveillance (AVSS), 12th IEEE International Conference on. IEEE, 2015

    In this paper we present a novel approach to detect people meeting. The proposed approach works by translating people behaviour from trajectory information into semantic terms. Having available a semantic model of the meeting behaviour, the event detection is performed in the semantic domain. The model is learnt employing a soft-computing clustering algorithm that combines trajectory information and motion semantic terms. A stable representation can be obtained from a series of examples. Results obtained on a series of videos with different types of meeting situations show that the proposed approach can learn a generic model that can effectively be applied on the behaviour recognition of meeting situations.
  • Multiresolution semantic activity characterisation and abnormality discovery in videos, L. Patino and J. Ferryman, Applied Soft Computing 2014

    This paper addresses the issue of activity understanding from video and its semantics-rich description. A novel approach is presented where activities are characterised and analysed at different resolutions. Semantic information is delivered according to the resolution at which the activity is observed. Furthermore, the multiresolution activity characterisation is exploited to detect abnormal activity. To achieve these system capabilities, the focus is given on context modelling by employing a soft computing-based algorithm which automatically enables the determination of the main activity zones of the observed scene by taking as input the trajectories of detected mobiles. Such areas are learnt at different resolutions (or granularities). In a second stage, learned zones are employed to extract people activities by relating mobile trajectories to the learned zones. In this way, the activity of a person can be summarised as the series of zones that the person has visited. Employing the inherent soft relation properties, the reported activities can be labelled with meaningful semantics. Depending on the granularity at which activity zones and mobile trajectories are considered, the semantic meaning of the activity shifts from broad interpretation to detailed description. Activity information at different resolutions is also employed to perform abnormal activity detection.
  • Temporally stable feature clusters for maritime object tracking, C. Osborne, T. Cane, T. Nawaz and J. Ferryman, in Advanced Video and Signal Based Surveillance (AVSS), 2015 12th IEEE International Conference on

    This paper describes a new approach to detect and track maritime objects in real time. The approach particularly addresses the highly dynamic maritime environment, panning cameras, target scale changes, and operates on both visible and thermal imagery. Object detection is based on agglomerative clustering of temporally stable features. Object extents are first determined based on persistence of detected features and their relative separation and motion attributes. An explicit cluster merging and splitting process handles object creation and separation. Stable object clusters are tracked frame-to-frame. The effectiveness of the approach is demonstrated on four challenging real-world public datasets.
  • A thermal object tracking benchmark, A. Berg, J. Ahlberg and M. Felsberg. In Advanced Video and Signal Based Surveillance (AVSS), 2015 12th IEEE International Conference on

    Short-term single-object (STSO) tracking in thermal images is a challenging problem relevant in a growing number of applications. In order to evaluate STSO tracking algorithms on visual imagery, there are de facto standard benchmarks. However, we argue that tracking in thermal imagery is different than in visual imagery, and that a separate benchmark is needed. The available thermal infrared datasets are few and the existing ones are not challenging for modern tracking algorithms. Therefore, we hereby propose a thermal infrared benchmark according to the Visual Visual Object Tracking (VOT) protocol for evaluation of STSO tracking methods. The benchmark includes the new LTIR dataset containing 20 thermal image sequences which have been collected from multiple sources and annotated in the format used in the VOT Challenge. In addition, we show that the ranking of different tracking principles differ between the visual and thermal benchmarks, confirming the need for the new benchmark.
  • Channel Coded Distribution Field Tracking for Thermal Infrared Imagery, A. Berg, J. Ahlberg and M. Felsberg. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 9-17

    We address short-term, single-object tracking, a topic that is currently seeing fast progress for visual video, for the case of thermal infrared (TIR) imagery. The fast progress has been possible thanks to the development of new template-based tracking methods with online template updates, methods which have not been explored for TIR tracking. Instead, tracking methods used for TIR are often subject to a number of constraints, e.g., warm objects, low spatial resolution, and static camera. As TIR cameras become less noisy and get higher resolution these constraints are less relevant, and for emerging civilian applications, e.g., surveillance and automotive safety, new tracking methods are needed. Due to the special characteristics of TIR imagery, we argue that template-based trackers based on distribution fields should have an advantage over trackers based on spatial structure features. In this paper, we propose a template-based tracking method (ABCD) designed specifically for TIR and not being restricted by any of the constraints above. In order to avoid background contamination of the object template, we propose to exploit background information for the online template update and to adaptively select the object region used for tracking. Moreover, we propose a novel method for estimating object scale change. The proposed tracker is evaluated on the VOT-TIR2015 and VOT2015 datasets using the VOT evaluation toolkit and a comparison of relative ranking of all common participating trackers in the challenges is provided. Further, the proposed tracker, ABCD, and the VOT-TIR2015 winner SRDCFir are evaluated on maritime data. Experimental results show that the ABCD tracker performs particularly well on thermal infrared sequences.
  • PETS 2016: Dataset and Challenge, L. Patino, T. Cane, Alain Vallee and J. Ferryman. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 18-25

    This paper describes the datasets and computer vision challenges that form part of the PETS 2016 workshop. PETS 2016 addresses the application of on-board multi sensor surveillance for protection of mobile critical assets. The sensors (visible and thermal cameras) are mounted on the asset itself and surveillance is performed around the asset. Two datasets are provided: (1) a multi sensor dataset as used for the PETS2014 challenge which addresses protection of trucks (the ARENA Dataset); and (2) a new dataset - the IPATCH Dataset - addressing the application of multi sensor surveillance to protect a vessel at sea from piracy. The dataset specifically addresses several vision challenges set in the PETS 2016 workshop, and corresponding to different steps in a video understanding system: Low-Level Video Analysis (object detection and tracking), Mid-Level Video Analysis (‘simple’ event detection: the behaviour recognition of a single actor) and High-Level Video Analysis (‘complex’ event detection: the behaviour and interaction recognition of several actors)
  • The IPATCH System for Maritime Surveillance and Piracy Threat Classification, M. Andersson, R. Johansson, K-G. Stenborg, R. Forsgren, T. Cane, G. Taberski, L. Patino and J. Ferryman. In EISIC 2016: 2016 European Intelligence and Security Informatics Conference

    In the EU FP7 project IPATCH, we are researching components for a maritime piracy early detection and avoidance system for deployment on merchant vessels. The system combines information from on-board sensors with intelligence from external sources in order to give early warnings about piracy threats. In this paper we present the ongoing work with the development of an integrated system fusing heterogeneous data.
  • A new Knowledgebase and Methodology for Analysing Piracy Incidents and Countermeasures, T. Cane. MAST Europe 2016 Conference, Amsterdam, Netherlands, 21-23 June 2016

    Modern maritime piracy, despite its apparent decline in recent years, represents a huge economic and human cost to the shipping industry. With the advent of BMP4 and other guidelines, ships have learned to defend themselves through various countermeasures, but inappropriate use can result in unnecessary cost and even place the ship and crew at greater risk. A better understanding of the effectiveness and implications of existing countermeasures is needed, as well as the legal and ethical issues surrounding their use. This has been studied as part of the European-funded research project, IPATCH (Intelligent Piracy Avoidance using Threat detection and Countermeasure Heuristics). The objectives of IPATCH were to collect, integrate and analyse historical data on piracy incidents and investigate the legal, ethical, economic and societal implications of countermeasures. The result of this work was the production of a piracy knowledgebase and a ‘manual’ for the shipping industry to support effective use of countermeasures. The knowledgebase was formed by fusing information from over 800 incident reports and complementary data from public sources into single database of 99 parameters. This was combined with a catalogue of countermeasures detailing their usage, costs and an assessment of legal and ethical implications. The knowledgebase is subsequently used for the calculation of piracy risk and countermeasure performance indicators in different situations, which are presented in the form of a ‘manual’. This paper explains the methodology used to collect, fuse and analyse the information on incidents and countermeasures and reports on the findings of the project.
  • An automated surveillance and decision support system to protect against piracy, T. Cane. MAST Europe 2016 Conference, Amsterdam, Netherlands, 21-23 June 2016

    Piracy is a constantly evolving threat. One thing that hasn’t changed is the need for a good lookout and constant awareness of the situation in order to give maximum early warning of a possible attack. However, crews are getting smaller and maintaining the watch is becoming more of a burden on seafarers. At the same time, commercial shipping cannot afford the kind of high-grade sensors used by the military. What is needed is an affordable technology solution to support the crew’s situational awareness. The proof-of-concept of such system is already being developed in the European-funded research project, IPATCH (Intelligent Piracy Avoidance using Threat detection and Countermeasure Heuristics), which runs until March 2017. IPATCH is developing an innovative on-board automated surveillance and decision support system which uses advanced software to compensate for the shortcomings of off-the-shelf sensor hardware. The IPATCH system combines low cost visual and thermal cameras with existing ship sensors (radar, AIS, navigation). The sensor data is fused and analysed to create the situational picture. Software algorithms then automatically assess the threats and provide critical information to the captain and crew so that they can make informed decisions. This paper explains how the data from multiple heterogeneous sensors is transformed into a common operational picture which is subsequently used for threat detection and decision support. The overall concept and architecture of the on-board system is presented, along with the results obtained in the IPATCH project.
  • A new dataset for maritime surveillance, T. Cane and L. Patino. MAST Europe 2016 Conference, Amsterdam, Netherlands, 21-23 June 2016

    Currently, there is a lack of publically available datasets for the scientific community and industry to use for the development and testing of maritime sensing and surveillance solutions. To address this, the IPATCH and AUTOPROTECTION projects have created a maritime surveillance dataset which consists of synchronised data from 13 visible and thermal cameras mounted on a vessel, plus radar, AIS and navigational sensors. The dataset was captured by acting out variations of 16 different scenarios around a vessel to generate interesting events for maritime surveillance applications. The scenarios are based on real-life piracy incidents, developed and refined by a panel of experts. They include ‘normal’, ‘suspicious’ and ‘threatening’ behaviours, involving varying numbers of targets (speedboats, ‘skiffs’, fishing boats, etc.). In total, the dataset consists of 59 sequences recorded by 17 sensors mounted on the vessel and is fully annotated to allow for development and performance evaluation of object detection, tracking and threat recognition algorithms. The dataset is unique in the sense it comprises a suite of heterogeneous sensors covering 360° around the vessel. This paper describes the sensors and scenarios included and the methodology used to generate the dataset. Results from the IPATCH project are also presented, to show how the dataset is being actively used to develop maritime surveillance solutions.
  • Situation assessment in the context of piracy threat, Ł. Kwieciński, M. Pokojski, G. Taberski, K-G. Stenborg and M. Andersson. MAST Europe 2016 Conference, Amsterdam, Netherlands, 21-23 June 2016

    Currently many vessels cross areas in which there is a risk of piracy attacks. Having this in mind a number of organizations, such as IMO, IMB or UKMTO, have issued special guidelines and provide help for those who might be attacked. On the basis of the past incidents we could draw a conclusion that in significant number of cases the pirate attack was detected too late to implement proper countermeasures. This implies that the captain’s situational awareness should be supported by solutions providing the common operational picture (COP) of the environment. The proof-of-concept of such system is already being developed in the FP7 IPATCH project (Intelligent Piracy Avoidance using Threat detection and Countermeasure Heuristics) founded by EC (grant number 607567). This system will include sensors deployed on the ship, tracking algorithms for the nearby vessels and specific modules, i.e. situation assessment, threat detection and decision support. This paper presents the concept the situation assessment module, being developed in IPATCH project, aimed at providing a captain with well-structured common operational picture. For this reason, dedicated situational features and relations, which strictly define specific situational characteristics, have been designed. The picture will be based on macro (using GIS data) and micro analyses of the situation (using ship tracks from AIS or cameras). The data model proposed within the paper represents comprehensive approach for maritime situation, which could be useful not only for IPATCH project, but also for other systems which need a holistic understanding of the maritime environment.
  • Semantic Modelling for Behaviour Characterisation and Threat Detection. by L. Patino, J. Ferryman; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 43-49

    Threat detection in computer vision can be achieved by extraction of behavioural cues. To achieve recognition of such cues, we propose to work with Semantic Models of behaviours. Semantic Models correspond to the translation of Low-Level information (tracking information) into High-Level semantic description. The model is then similar to a naturally spoken description of the event. We have built semantic models for the behaviours and threats addressed in the PETS 2016 IPATCH dataset. Semantic models can trigger a threat alarm by themselves or give situation awareness. We describe in this paper how semantic models are built from Low-Level trajectory features and how they are recognised. The current results are promising.
  • Saliency-Based Detection for Maritime Object Tracking, by T. Cane, J. Ferryman; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 18-25

    This paper presents a new method for object detection and tracking based on visual saliency as a way of mitigating against challenges present in maritime environments. Object detection is based on adaptive hysteresis thresholding of a saliency map generated with a modified version of the Boolean Map Saliency (BMS) approach. We show that the modification reduces false positives by suppressing detection of wakes and surface glint. Tracking is performed by matching detections frame to frame and smoothing trajectories with a Kalman filter. The proposed approach is evaluated on the PETS 2016 challenge dataset on detecting and tracking boats around a vessel at sea.
  • Threat detection and situational awareness for merchant shipping, by Andersson, M., Wadströmer, N., Stenborg, K-G., Petersson, H., Forsgren, R., Allvar, J., TAMSEC 2015

    In the EU FP7 project IPATCH1 we are researching components for an early piracy detection and avoidance system deployed on merchant ships. The system combines information from onboard sensors about the local situation with intelligence from other sources about the current situation in the region in order to give early warning about threatening pirates.
  • From sensor data to piracy threat classification for merchant shipping, by Stenborg, K-G., Andersson, M., Allvar, J., Johansson R, Wadströmer, N., SSBA 2016

    In the EU FP7 project IPATCH, we are researching components for an early piracy detection and avoidance system deployed on merchant ships. The system combines information from on-board sensors about the local situation with intelligence from other sources about the current situation in the region in order to give early warning about threatening pirates. One of the challenges in the project is how data from the on-board sensors should be derived and represented as motion indicators, that can be fused with text and numerical values from other sources, for threat classification and situation awareness. In this paper we describe our approach with Bayesian networks, which are used for fusing different types of indicators. Motion indicators are based on detection and tracking data from different sensors.
  • Le Corps ne ment pas : Biopolitiques Postmodernes et Biométrie ("The body doesn’t lie: mapping postmodern biopolitics and biometry")

    Presentation by Nathalie Grandjean (Université de Namur) at an interdisciplinerary seminar organised at the Institut Français de l’Éducation de l’École Normale Supérieure de Lyon.
  • The IPATCH Dataset: A comprehensive maritime benchmark for detection, tracking and threat recognition

    This paper describes a new multimodal maritime dataset recorded using a multispectral suite of sensors, including AIS, GPS, radar, and visible and thermal cameras. To be presented at CVPR 2017, which will take place at the Hawaii Convention Center from July 21 to July 26, 2017 in Honolulu, Hawaii.

    Uploaded:  25 / May / 2017

  • PETS 2017: Dataset and Challenge

    This paper indicates the dataset and challenges evaluated under PETS2017. This year, the BMTT Tracking Challenge (Benchmarking of Multi-Target Tracking) and PETS (Performance Evaluation of Tracking and Surveillance) have joined to organise the first BMTT-PETS workshop of tracking and surveillance, in conjunction with CVPR 2017.

    Uploaded:  25 / May / 2017

  • Loitering Behaviour Detection of Boats at Sea

    We present in this paper a technique for Loitering detection based on the analysis of activity zones of the monitored area. Activity zones are learnt online employing a soft computing-based algorithm which takes as input the trajectory of object mobiles appearing on the scene. Statistical properties on zone occupancy and transition between zones makes it possible to discover abnormalities without the need to learn abnormal models beforehand. We have applied this approch to the PETS2017 IPATCH dataset and addressed the challenge on detecting skiff boats loitering around a protected ship, which eventually is attacked by the skiffs. Our results show that we can detect the suspicious behaviour on time to trigger an early warning

    Uploaded:  2 / Aug / 2017