RF-based device-free recognition
We consider the detection of activities of device-free entities from the analysis of RF-channel fluctuations induced by these very activities. In analogy to the definition of device-free radio-based localisation systems (DFL) we define device-free radio-based activity recognition systems (DFAR) as systems which recognise the activity of a person using analysis of radio signals while the person itself is not required to carry a wireless device. Device-free radio-based recognition can extend the awareness of wireless-enabled devices over the device boundaries. It might be applied e.g. for devices in the upcoming IoT as well as for mobile phones, laptop computers or consumer products incorporating an interface to the wireless channel.
|Sensor type||Device-free; device-bound|
|Sensing modality||Passive; active|
|Setup||Ad-hoc, non ad-hoc (requires training)|
|Sensed value||Continuous signal (e.g. SDR); Channel-Quality-Indicator (e.g. RSSI)|
Most work conducted in the area of RF-based classification with passive participants is related to the localisation of individuals. The feasibility of this approach was verified in various environmental settings and at various frequencies. The features utilised are mostly the RSSI, its moving average, mean or RSSI fingerprint. Also, 2-way RSSI variance was employed. With these features a localisation accuracy of about 0.5 meters was possible or the simultaneous localisation of up to 5 persons in a changing environment with an accuracy of 1 meter. While the localisation of individuals based on features from the radio channel can therefore generally be considered as solved, recently, some authors considered active DFAR approaches to also detect activities. DFAR is still a mostly unexplored research field. Open research questions regard the optimum frequencies and the impact of the frequency on the classification accuracy, the optimum sampling rate of the signal, the detection range and the impact of this distance on the classification accuracy as well as the minimum Signal-to-Noise Ratio (SNR). Furthermore, a set of activities that can be recognised by RF-based classification is yet to be identified as well as a suitable design of the detection system. In particular, the impact of the count and height of transmitting and receiving nodes has not yet been considered comprehensively as well as even the actual necessity of a transmit node as part of the recognition system since potentially the system might utilise ambient radio. Also, it is not clear whether and how activities of multiple persons can be identified simultaneously and if features exist that enable ad-hoc DFAR systems. A more detailed discussion of most of these aspects is given by Scholz et al.
- Formal definition of RF-based Device-Free Activity-Recognition (DFAR)
- Discussion of features for the recognition of activities in active and passive (both RSSI and SDR-based) device-free RF-based systems
- Focus on time-domain features for an easy integration in consumer-hardware
- Case studies and demonstrators for the recognition of activities in active and passive (both RSSI and SDR-based) device-free RF-based systems
- Monitoring of attention with a passive Rf-based DFAR systems
- Classification of simple actions (walking, crawling, lying, standing, sitting) with active and passive (both RSSI and SDR-based) device-free RF-based systems
- Recognition of gestures in active SDR-based and passive RSSI-based systems
- Distinction of walking speed with active and passive (both RSSI and SDR-based) device-free RF-based systems
- Distinction of 11 gestures with a passive RSSI-based device-free RF-based system (The first ever implementation on an off-the-shelf mobile phone)
Active; continuous signal
We have proposed a non-ad-hoc, active device-free activity recognition system. For this system, we derived feasible features to detect simple activities such as walking, crawling, standing, lying. For these activities we could reach a classification accuracy that is comparable to accelerometer (device-bound) based recognition systems (compared to intertion recognition system in the same scenario). Classification was achieved by k-NN, Naive Bayes and decision tree classifiers with similar classification accuracy. The presented work is the first to detect the considered activities from RF-channel measurements and also the first to do this with active and passive DFAR systems. Despite some recent advances on device-free radio-based localisation systems, this is also the first study to combine an activity recognition and localisation in one classification algorithm on a common set of features. For the activities lying, crawling, standing and walking we were able to localise them within less than 1 meter in USRP-SDR-based case-studies with the active DFAR system. The results of this study effectively enable the use of arbitrary wireless devices as sensing equipment. Furthermore, we have considered the use of environmental RF-sensors to distinguish these basic activities in an indoor environment conducted simultaneously by two subjects not equipped with an antenna. Generally, and not surprisingly, the recognition accuracy increases when more sensors are utilised. We achieved good results with three or more receive nodes (see the table on the right where four receive nodes have been utilised). Then, the 25 distinct classes of combinations of these simultaneously conducted activities could be well distinguished.
We compared the classification accuracy of active and passive SDR-based systems and also to recognition based on accelerometer devices (device-bound) to the accuracy achieved with an active RSSI-based system implemented through sensor nodes deployed in an indoor environment. With only a single transmitter and receiver (exchanging packets at 100 Hz), the above basic activity cases could be well distinguished. However, the accuracy acheived is lower than for SDR-based and accelerometer-based system. The main challenge in an RSSI-based system is that the granularity and sample rate of RSSI is lower than for the SDR-based system. As a consequence, only time-domain features are feasible.
Passive; continuous signal
The above mentioned active approaches reqire a transmitter as part of the recognition system. These systems can reach best performances since the signal used for recognition is also under the control of the system. However, the deployment effort and cost are higher for such kind of approaches. As an alternative, we investigated passive SDR-based DFAR systems in which signals from environmental FM radio stations are leveraged for the recognition of activities. With this deployment and a single receiver, it was possible to reach similar accuracies as for the active SDR-based system. Also, we investigated the potential to measure attention of persons towards electronic poster frames that have the antenna integrated in their frame. We combined the detection of walking speed and location in order to distinguish attention levels.
We investigated the use of WiFi Received Signal Strength Information (RSSI) at a mobile phone for the recognition of situations, activities and gestures. In particular, we propose a passive activity recognition system that does not require any device carried by the user. We discussed challenges and lessons learned for the design of such a system on a mobile phone and propose appropriate features for extracting activity characteristics from RSSI signals. We demonstrate the feasibility of recognising activities, gestures and environmental situations from RSSI information obtained by the mobile phone. The case studies were conducted during a period of about two months in which about 12 hours of continuous RSSI data has been sampled, in two countries and with 11 participants in total. Results demonstrate the potential to utilise RSSI information for the extension of the environmental perception of a mobile device as well as for the interaction with touch-free gestures. The system achieves an accuracy of 0.51 while distinguishing as many as 11 gestures and can reach 0.72 on average for four more disparate ones.
- WLAN-Router werden zum Raumüberwacher (Saarbruecker Zeitung, 01.09.2014) (PDF)
- 4th CoSDEO workshop on Device-free Radio-based recognition
- Sensewaves Video on Youtube
- Bewegungserkennung per Funk (ZeitOnline, 17.03.2014)
- WLAN-Signale verraten, was Menschen tun Bewegungserkennung per Funk (golem.de, 17.03.2014)
- Ueberwachung: Bewegungserkennung per Funk (Heise Online, 18.03.2014)
- Die neue Form der Überwachung (Radio Eins RBB, 29.03.2014)
- Stephan Sigg, Ulf Blanke and Gerhard Troester: The Telepathic Phone: Frictionless Activity Recognition from WiFi-RSSI, accepted by the IEEE International Conference on Pervasive Computing and Communications (PerCom), Budapest, Hungary, March 24-28, 2014
- Shuyu Shi, Stephan Sigg, Wei Zhao, and Yusheng Ji: Monitoring of Attention from Ambient FM-radio Signals, accepted for publication in IEEE Pervasive Computing, Special Issue - Managing Attention in Pervasive Environments, Los Alamitos, CA, USA, IEEE Computer Society, 2014
- Stephan Sigg, Mario Hock, Markus Scholz, Gerhard Troester, Lars Wolf, Yusheng Ji and Michael Beigl: Passive, device-free recognition on your mobile phone: tools, features and a case study, 10th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (Mobiquitous 2013), Tokyo, Japan, 2013
- Stephan Sigg, Shuyu Shi, Felix Buesching, Yusheng Ji and Lars Wolf: Leveraging RF-channel fluctuation for activity recognition, 11th International Conference on Advances in Mobile Computing and Multimedia (MoMM2013), Vienna, Austria, 2013
- Stephan Sigg, Shuyu Shi, and Yusheng Ji: RF-based device-free recognition of simultaneously conducted activities, adjunct proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013), Zurich, Switzerland, 2013
- Shuyu Shi, Stephan Sigg and Yusheng Ji: Joint Localisation and Activity Recognition from Ambient FM Broadcast Signals, adjunct proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp 2013), Zurich, Switzerland, 2013
- Shuyu Shi, Stephan Sigg and Yusheng Ji: ActiviTune: A Multi-stage system for activity recognition of passive entities from ambient FM-radio signals, in the 8th International Conference on Wireless Algorithms, Systems, and Applications (WASA 2013)
- Stephan Sigg, Markus Scholz, Shuyu Si, Yusheng Ji and Michael Beigl: RF-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals, in IEEE Transactions on Mobile Computing (TMC), Feb. 2013, vol. 13, no. 4 (DOI)
- Shuyu Shi, Stephan Sigg and Yusheng Ji: A comparison of two approaches to activity recognition of individuals in an indoor environment, in 2013 IEICE General Conference
- Shuyu Shi, Stephan Sigg and Yusheng Ji: Passive detection of situations from ambient FM-radio signals, in Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp '12). Pittsburgh, PA, USA, 1049-1053, 2012 (doi: 10.1145/2370216.2370440) (DOI)
- Shuyu Shi, Stephan Sigg and Yusheng Ji: Activity recognition from Radio Frequency data: Two-stage recognition and features, in Proceedings of the 2012 IEICE Society Conference
- Shuyu Shi, Stephan Sigg and Yusheng Ji: Activity recognition from Radio Frequency data: Multi-stage recognition and features, in Proceedings of the 2012 IEEE Vehicular Technology Conference (VTC Fall), pp.1-6, 3-6 Sept. 2012 (DOI)
- Stephan Sigg, Lei Zhong, Yusheng Ji: Activity recognition with implicit context classification, in Proceedings of the 2012 IEEE Vehicular Technology Conference (VTC Fall), pp.1-6, 3-6 Sept. 2012 (DOI)
- Markus Scholz, Stephan Sigg, Hedda Schmidtke and Michael Beigl: Challenges for device-free radio-based activity recognition, in Proceedings of the 3rd Workshop on Context Systems Design Evaluation and Optimisation (CoSDEO), 2011 (PDF)
- Markus Scholz, Stephan Sigg, Dimana Shihskova, Georg von Zengen, Gerrit Bagshik, Toni Guenther, Michael Beigl and Yusheng Ji: SenseWaves: Radiowaves for context recognition, in Video Proceedings of the 9th International Conference on Pervasive Computing (Pervasive 2011), 2011 (Video, PDF)
- Johannes Starosta, Markus Reschke, Sebastian Schwarzl, Stephan Sigg and Michael Beigl: Context awareness through the RF-channel, in 2nd workshop for Context Systems Design and Evaluation (CoSDEO) in conjunction with 5th Conference on Architecture for Computing Systems (ARCS), 2011 (PDF)
- Stephan Sigg, Michael Beigl and Behnam Banitalebi: Efficient adaptive communication from resource restricted transmitters, In Organic Computing - A Paradigm Shift for Complex Systems, Christian Mueller-Schloer and Hartmut Schmeck and and Theo Ungerer, Springer, 2011 (Online)
- Stephan Sigg and Michael Beigl: An adaptive protocol for distributed beamforming, in 17. Fachtagung 'Kommunikation in Verteilten Systemen 2011' (KiVS11), 2011 (PDF)
- Markus Reschke, Johannes Starosta, Sebastian Schwarzl and Stephan Sigg: Situation awareness based on channel measurements, 2011 73rd IEEE Vehicular Technology Conference (VTC Spring), pp.1-5, 15-18 May 2011 (DOI)
- Stephan Sigg and Michael Beigl: Distributed adaptive expectation aware in-network context processing, in Fourth ACM International Workshop on Context-Awareness for Self-Managing Systems, 2010 (DOI, slides)