Tag Archive: total information awareness

N.S.A. Gathers Data on Social Connections of U.S. Citizens

nsa_eyeFrom New York Times

WASHINGTON — Since 2010, the National Security Agency has been exploiting its huge collections of data to create sophisticated graphs of some Americans’ social connections that can identify their associates, their locations at certain times, their traveling companions and other personal information, according to newly disclosed documents and interviews with officials.

Gen. Keith Alexander, the director of the National Security Agency, testified on Thursday before the Senate Intelligence Committee.

The spy agency began allowing the analysis of phone call and e-mail logs in November 2010 to examine Americans’ networks of associations for foreign intelligence purposes after N.S.A. officials lifted restrictions on the practice, according to documents provided by Edward J. Snowden, the former N.S.A. contractor.

The policy shift was intended to help the agency “discover and track” connections between intelligence targets overseas and people in the United States, according to an N.S.A. memorandum from January 2011. The agency was authorized to conduct “large-scale graph analysis on very large sets of communications metadata without having to check foreignness” of every e-mail address, phone number or other identifier, the document said. Because of concerns about infringing on the privacy of American citizens, the computer analysis of such data had previously been permitted only for foreigners.

The agency can augment the communications data with material from public, commercial and other sources, including bank codes, insurance information, Facebook profiles, passenger manifests, voter registration rolls and GPS location information, as well as property records and unspecified tax data, according to the documents. They do not indicate any restrictions on the use of such “enrichment” data, and several former senior Obama administration officials said the agency drew on it for both Americans and foreigners. (more…)

How the cops watch your tweets in real-time

bluejay2From Arstechnica

Recent leaks about the NSA’s Internet spy programs have sparked renewed interest in government surveillance, though the leaks touch largely on a single form of such surveillance—the covert one. But so-called “open source intelligence” (OSINT) is also big business— and not just at the national/international level. New tools now mine everything from “the deep Web” to Facebook posts to tweets so that cops and corporations can see what locals are saying. Due to the sheer scale of social media posts, many tools don’t even aim at providing a complete picture. Others do.

For instance, consider BlueJay, the “Law Enforcement Twitter Crime Scanner,” which provides real-time, geo-fenced access to every single public tweet so that local police can keep tabs on #gunfire, #meth, and #protest (yes, those are real examples) in their communities. BlueJay is the product of BrightPlanet, whose tagline is “Deep Web Intelligence” and whose board is populated with people like Admiral John Poindexter of Total Information Awareness infamy.

BlueJay allows users to enter a set of Twitter accounts, keywords, and locations to scan for within 25-mile geofences (BlueJay users can create up to five such fences), then it returns all matching tweets in real-time. If the tweets come with GPS locations, they are plotted on a map. The product can also export databases of up to 100,000 matching tweets at a time.

A look at the BlueJay interface shows it to be a fairly basic tool, but one that gets its power from full access to the Twitter “firehose” of all tweets. Users who want to search the Twitterverse have three basic options: Twitter’s search API, Twitter’s streaming API, or full firehouse access to Twitter from third party providers like GNIP and DataSift. The first two are free but limited; BrightPlanet notes that even the broader streaming API returns somewhere between one and 40 percent of the relevant tweets depending on Twitter’s load at the moment. The firehose requires some serious infrastructure and a paid contract with Twitter, but it provides all relevant tweets. (more…)