The Pentagon has developed a new search engine that can reach into the deep, dark recesses of the World Wide Web
For more than a year now, the Defense Advanced Research Projects Agency (DARPA) has been working on Memex, a search engine that can gather information from the 90 to 95 percent of the Internet that Google's search crawlers cannot reach.
The search engine, which gets its name from a combination of "memory" and "index," was designed to explore the unchartered terrains of the Internet, including the databases of information not indexed by Google and the darker part of the web where criminal activities, such as sex trafficking and drug dealing, take place.
"We're envisioning a new paradigm for search that would tailor content, search results, and interface tools to individual users and specific subject areas, and not the other way around," said DARPA program manager Chris White in a statement. "By inventing better methods for interacting with and sharing information, we want to improve search for everybody and individualize access to information. Ease of use for non-programmers is essential."
With a team of 17 contractors, DARPA developed Memex far enough that the tool has become essential in the conviction of a New York City man who was part of a group who kidnapped and sexually assaulted a 28-year-old woman before she escaped through the window of the six-story building where she was held. Manhattan District Attorney Cyrus R. Vance says his office now uses Memex for every single human trafficking case it works on, which is the initial intention for the search engine.
Unlike the search results offered by Google, Bing, and Yahoo, which are based on advertising and displayed in a linear fashion based on the search engine's algorithms, Memex finds information that is difficult to find on Google and serves them as sophisticated infographics showing the vast network of relationships of the information.
For instance, searching for a name and phone number that crop up in a sex trafficking ad would not bring up a list of other places on the web where the name and number show up. Instead, Memex would create a diagram containing dots representing the web pages containing the name and number, thus drawing a bigger picture of what could possibly be a human trafficking ring operating online.
Clicking on the dots would show the place and time where the ads were posted, making it possible for criminal investigation bodies to go after criminals they would otherwise not be able to locate using a basic Google search.
The nature of Memex certainly raises questions from privacy advocates, but DARPA is quick to emphasize that it is "specifically not interested" in accessing information that is not publicly available online or identifying anonymous services, servers or IP addresses.
Aside from going after human traffickers, terrorists and other criminals, DARPA says Memex can become particularly useful to government, military, and commercial organizations in finding and organizing "mission critical" information on the Internet. Emergency responders, for example, can quickly find information on the worst hit areas in the event of a natural disaster.
