Using Twitter, Smartphones To Accurately Assess Crowds - InformationWeek
IoT
IoT
Software // Enterprise Applications
Commentary
5/29/2015
08:06 AM
100%
0%

Using Twitter, Smartphones To Accurately Assess Crowds

As part of the Big Data Challenge, researchers from England's University of Warwick have turned to smartphones and Twitter to estimate crowd sizes quickly and accurately.

7 Bold Tech Ideas That Will Make You Uncomfortable
7 Bold Tech Ideas That Will Make You Uncomfortable
(Click image for larger view and slideshow.)

Whenever there's a crowd there will be estimates of how many people were there. History shows that those estimates tend to be highly variable and terribly inaccurate. Now, researchers from England's University of Warwick have turned to smartphones and Twitter to estimate crowd sizes quickly and accurately.

This has obvious implications for law enforcement and other government agencies, but is also valuable for any enterprise interested in assessing crowd volume. For example, US retailers could use such a solution to obtain an accurate assessment of post-Thanksgiving Black Friday shopping crowds. Sports teams and concert promoters could make use of these tools to gauge activity during events. And transit systems could make use of such data to understand commuter travel patterns.

The research, conducted by Federico Botta, PhD student; Suzy Moat, assistant professor of Behavioral Science; and Tobias Preis, associate professor of Behavioral Science and Finance at University of Warwick's Warwick Business school, has broad implications because it doesn't depend on reporting from a special app that sits on users' smartphones. It looks, instead, at activity from those smartphones, and the number of smartphones in use in a given area, to estimate crowd size and even predict future activity. Because these methods don't require identifying individual users or looking at the contents of their activities, they shouldn't require as much judicial approval as other, more intrusive techniques.

In a telephone interview with the three researchers, Preis said, "We thought that people would be walking around with smartphones in their pockets, and even if they're not using them they would be connected to the Internet. We thought we could estimate the crowd size accurately and very quickly." He further explained, "Twitter was one of the components we looked at. We also used the volume of phone calls and SMS calls, as well as the volume of access to the Internet."

(Image: JESHOOTS via Pixabay)

(Image: JESHOOTS via Pixabay)

Accuracy in estimating the size comes from correlating data from these multiple sources. When asked about access to call and SMS data, the three explained that their data was made available by the Italian telecom company Telecom Italia as part of the 2014 Big Data Challenge. For the challenge, the company made multiple data sets available for researchers and software developers to work with.

[ Curious about how crowds can be used in software development? Read Software Development Taps The Power Of Crowds. ]

One of the challenge's data sets involved smartphone use in and around a sports stadium during a football (soccer) match. Preis said, "We focused on a football stadium. This was a great test case because the number of people inside is well known." The known crowd size meant that the researchers were able to test their theories and algorithms and check the accuracy against a known answer.

As for the models and algorithms themselves, Preis said, "The analysis was stunningly straightforward." Explaining that most of the software the researchers used is available through open source licenses, Preis said, "We're using the statistical package 'R' and extending the analytics. We also wrote customized tools and programs in R and C, depending on the complexity and the run-time requirements."

Because the team members based their work on multiple data sources and types, they weren't able to use a single program for the entire project. "C, Python, and other interfaces were used. Obviously, we needed to work with a lot of APIs, depending on the data source," Preis said. As for the hardware involved, Botta said that most software and services were run on commodity Intel hardware, though some of the more compute-intensive routines were hosted on the GPU-based systems that now make up a large percentage of high-performance computing platforms.

As for the future, Moat said that publication of these results doesn't mark the end of the work. "The research we presented is part of a larger body of research," she said. "We're trying to see if we can use online data to analyze and predict what people are doing in the real world. In the past we've used data from Google and Wikipedia to look at movements in the stock market, and in epidemiology such as flu epidemics," she explained. Now, "... we're working in crime and crowd size estimation," Moat said.

Those interesting in similar work should take note of the 2015 Big Data Challenge, which is now open.

[Did you miss any of the InformationWeek Conference in Las Vegas last month? Don't worry: We have you covered. Check out what our speakers had to say and see tweets from the show. Let's keep the conversation going.]

Curtis Franklin Jr. is Senior Editor at Dark Reading. In this role he focuses on product and technology coverage for the publication. In addition he works on audio and video programming for Dark Reading and contributes to activities at Interop ITX, Black Hat, INsecurity, and ... View Full Bio
We welcome your comments on this topic on our social media channels, or [contact us directly] with questions about the site.
Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
Page 1 / 3   >   >>
PedroGonzales
50%
50%
PedroGonzales,
User Rank: Ninja
6/13/2015 | 7:58:32 PM
Re: Legal issues?
@ CurtFranklin. what sometimes dissapoints me about following debates about the impact of new technology into our lives is that. The debate continues on and one but there isn't an end. I would like hear that people debate on an issue and then produce some sort of conclusion about the issue which could can be implemented.  By the time the debate finishes the technology would have change completely.
Curt Franklin
50%
50%
Curt Franklin,
User Rank: Strategist
6/9/2015 | 10:58:35 AM
Re: Legal issues?
@PedroGonzales, that's always the trade-off: Is the technology's use to the authorities for legitimate purposes of sufficient value to risk possible erosion of privacy rights? That's where the debate is going to be happening for a long time to come, I fear.
PedroGonzales
50%
50%
PedroGonzales,
User Rank: Ninja
6/4/2015 | 10:06:01 PM
Re: Legal issues?
I think law enforcement will be able to manage protest here in the states if they have access to such records.  The ACLU wouldn't be very happy. If it helps reduce vandalism such as what we saw in Baltimore, I think it will be a good approach.
Ariella
50%
50%
Ariella,
User Rank: Author
6/4/2015 | 5:08:34 PM
Re: Legal issues?
That's the only thing that one can be sure about, Curtis! Forces will be pulling both ways. 
Curt Franklin
50%
50%
Curt Franklin,
User Rank: Strategist
6/4/2015 | 4:03:02 PM
Re: Legal issues?
@Ariella, you're right -- there will be more pressure for US regulation. As we enter into another election cycle it's going to be interesting to hear how the various candidates couch their support for more or less regulation and more or less surveillance.

It's going to be a bumpy couple of years.
Curt Franklin
50%
50%
Curt Franklin,
User Rank: Strategist
6/4/2015 | 3:54:31 PM
Re: Legal issues?
@jastroff, I'm afraid Im feeling a bit cynical these days -- I can't see any particular agency that will take on the challenge because, honestly, there's too much money in allowing the situation to continue as it is.
Ariella
50%
50%
Ariella,
User Rank: Author
6/4/2015 | 2:18:24 PM
Re: Legal issues?
@Curt, indeed. The US has more of a tradition of attempting to allow people more freedom and for industries to be more self-governing. But with the global connections we all have, there is more pressure to comply with standards set abroad, and that, I would expect, will increase the pressure in the US for more governement regulation. 
jastroff
50%
50%
jastroff,
User Rank: Ninja
6/4/2015 | 2:15:48 PM
Re: Legal issues?
Yes, agree. But which group/agency - foreign or domestic, wants to/will take this one on...any thoughts?

<<  i think most people would be appalled if they realized just how much they give up in most EULAs.
Curt Franklin
50%
50%
Curt Franklin,
User Rank: Strategist
6/4/2015 | 2:10:42 PM
Re: Legal issues?
@Ariella, it's interesting to see the ebb and flow of practices that are considered rational or too intrusive in various nations. I fear the trend in both Europe and North America is toward more intrusive surveillance, always introduced with the most benign rationale. It will be most fascinating to see whether the general population ever decides that things have gone too far -- and whether, at that point, they can do anything about it.
Ariella
50%
50%
Ariella,
User Rank: Author
6/4/2015 | 2:04:22 PM
Re: Legal issues?
@Curtis that fits with the greater governmental control in Europe, though. They are much more inclined to nanny state practices, though the US is following in their footsteps. 
Page 1 / 3   >   >>
Register for InformationWeek Newsletters
White Papers
Current Issue
Cybersecurity Strategies for the Digital Era
At its core, digital business relies on strong security practices. In addition, leveraging security intelligence and integrating security with operations and developer teams can help organizations push the boundaries of innovation.
Video
Slideshows
Twitter Feed
Sponsored Live Streaming Video
Everything You've Been Told About Mobility Is Wrong
Attend this video symposium with Sean Wisdom, Global Director of Mobility Solutions, and learn about how you can harness powerful new products to mobilize your business potential.
Flash Poll