Sorry Noonan but Wired magazine has reported per/ NOW on PBS/ who provides links to story and evidence /the NSA is working with AT&T who have rerouted all emails to a room in AT&T that has NSA accesiblity only. It appears NSA has another room in MAE WEST the Western part of the state routed into San Fransisco also.
I am including two articles from wired and the AT&T traffic engineering document that allows AT&T to do this.

1st wired article- titled Whistle-Blower Outs NSA Spy Room
Wired News on the Go <http://www.wired.com/support/members.html>

By Ryan Singel Whistle-Blower Outs NSA Spy Room&story by this reporter
11:15 AM Apr, 07, 2006

AT&T provided National Security Agency eavesdroppers with full access to its customers' phone calls, and shunted its customers' internet traffic to data-mining equipment installed in a secret room in its San Francisco switching center, according to a former AT&T worker cooperating in the Electronic Frontier Foundation's lawsuit against the company.

Mark Klein, a retired AT&T communications technician, submitted an affidavit in support of the EFF's lawsuit this week. That class action lawsuit, filed in federal court in San Francisco last January, alleges that AT&T violated federal and state laws by surreptitiously allowing the government to monitor phone and internet communications of AT&T customers without warrants.

On Wednesday, the EFF asked the court to issue an injunction prohibiting AT&T from continuing the alleged wiretapping, and filed a number of documents under seal, including three AT&T documents that purportedly explain how the wiretapping system works.

According to a statement released by Klein's attorney, an NSA agent showed up at the San Francisco switching center in 2002 to interview a management-level technician for a special job. In January 2003, Klein observed a new room being built adjacent to the room housing AT&T's #4ESS switching equipment, which is responsible for routing long distance and international calls.

"I learned that the person whom the NSA interviewed for the secret job was the person working to install equipment in this room," Klein wrote. "The regular technician work force was not allowed in the room."

Klein's job eventually included connecting internet circuits to a splitting cabinet that led to the secret room. During the course of that work, he learned from a co-worker that similar cabinets were being installed in other cities, including Seattle, San Jose, Los Angeles and San Diego.

"While doing my job, I learned that fiber optic cables from the secret room were tapping into the Worldnet (AT&T's internet service) circuits by splitting off a portion of the light signal," Klein wrote.

The split circuits included traffic from peering links connecting to other internet backbone providers, meaning that AT&T was also diverting traffic routed from its network to or from other domestic and international providers, according to Klein's statement.

The secret room also included data-mining equipment called a Narus STA 6400, "known to be used particularly by government intelligence agencies because of its ability to sift through large amounts of data looking for preprogrammed targets," according to Klein's statement.

NARUS whose website touts AT&T as a client, sells software to help internet service providers and telecoms monitor and manage their networks, look for intrusions, and wiretap phone calls as mandated by federal law.

Klein said he came forward because he does not believe that the Bush administration is being truthful about the extent of its extrajudicial monitoring of Americans' communications.

"Despite what we are hearing, and considering the public track record of this administration, I simply do not believe their claims that the NSA's spying program is really limited to foreign communications or is otherwise consistent with the NSA's charter or with FISA," Klein's wrote. "And unlike the controversy over targeted wiretaps of individuals' phone calls, this potential spying appears to be applied wholesale to all sorts of internet communications of countless citizens."

After asking for a preview copy of the documents last week, the government did not object to the EFF filing the paper under seal, although the EFF asked the court Wednesday to make the documents public.

One of the documents is titled "Study Group 3, LGX/Splitter Wiring, San Francisco," and is dated 2002. The others are allegedly a design document instructing technicians how to wire up the taps, and a document that describes the equipment installed in the secret room.

In a letter to the EFF, AT&T objected to the filing of the documents in any manner, saying that they contain sensitive trade secrets and could be "could be used to 'hack' into the AT&T network, compromising its integrity."

According to court rules, AT&T has until Thursday to file a motion to keep the documents sealed. The government could also step in to the case and request that the documents not be made public, or even that the entire lawsuit be barred under the seldom-used

AT&T spokesman Walt Sharp declined to comment on the allegations, citing a company policy of not commenting on litigation or matters of national security, but did say that "AT&T follows all laws following requests for assistance from government authorities."

2nd Wired article- titled
Whistle-Blower's Evidence, Uncut02:00 AM May, 22, 2006

Former AT&T technician Mark Klein is the key witness in the Electronic Frontier Foundation's class-action lawsuit against the telecommunications company, which alleges that AT&T cooperated in an illegal National Security Agency domestic surveillance program.

Inside the Secret Room




In a public statement Klein issued last month, he described the NSA's visit to an AT&T office. In an older, less-public statement recently acquired by Wired News, Klein goes into additional details of his discovery of an alleged surveillance operation in an AT&T building in San Francisco.

Klein supports his claim by attaching excerpts of three internal company documents: a Dec. 10, 2002, manual titled "Study Group 3, LGX/Splitter Wiring, San Francisco," a Jan. 13, 2003, titled "SIMS, Splitter Cut-In and Test Procedure" and a second "Cut-In and Test Procedure" dated Jan. 24, 2003.

Here we present Klein's statement in its entirety, with inline links to all of the document excerpts where he cited them. You can also download the (pdf). The full AT&T documents are filed under seal in federal court in San Francisco.



AT&T's Implementation of NSA Spying on American Citizens

31 December 2005

I wrote the following document in 2004 when it became clear to me that AT&T, at the behest of the National Security Agency, had illegally installed secret computer gear designed to spy on internet traffic. At the time I thought this was an outgrowth of the notorious Total Information Awareness program, which was attacked by defenders of civil liberties. But now it's been revealed by The New York Times that the spying program is vastly bigger and was directly authorized by President Bush, as he himself has now admitted, in flagrant violation of specific statutes and constitutional protections for civil liberties. I am presenting this information to facilitate the dismantling of this dangerous Orwellian project.



AT&T Deploys Government Spy Gear on WorldNet Network

-- 16 January, 2004

In 2003 AT&T built "secret rooms" hidden deep in the bowels of its central offices in various cities, housing computer gear for a government spy operation which taps into the company's popular WorldNet service and the entire internet. These installations enable the government to look at every individual message on the internet and analyze exactly what people are doing. Documents showing the hardwire installation in San Francisco suggest that there are similar locations being installed in numerous other cities.

The physical arrangement, the timing of its construction, the government-imposed secrecy surrounding it and other factors all strongly suggest that its origins are rooted in the Defense Department's Total Information Awareness (TIA) program which brought forth vigorous protests from defenders of constitutionally protected civil liberties last year:

"As the director of the effort, Vice Adm. John M. Poindexter, has described the system in Pentagon documents and in speeches, it will provide intelligence analysts and law enforcement officials with instant access to information from internet mail and calling records to credit card and banking transactions and travel documents, without a search warrant." The New York Times, 9 November 2002

To mollify critics, the Defense Advanced Research Projects Agency (Darpa) spokesmen have repeatedly asserted that they are only conducting "research" using "artificial synthetic data" or information from "normal DOD intelligence channels" and hence there are "no U.S. citizen privacy implications" (Department of Defense, Office of the Inspector General report on TIA, December 12, 2003). They also changed the name of the program to "Terrorism Information Awareness" to make it more politically palatable. But feeling the heat, Congress made a big show of allegedly cutting off funding for TIA in late 2003, and the political fallout resulted in Adm. Poindexter's abrupt resignation last August. However, the fine print reveals that Congress eliminated funding only for "the majority of the TIA components," allowing several "components" to continue (DOD, ibid). The essential hardware elements of a TIA-type spy program are being surreptitiously slipped into "real world" telecommunications offices.

In San Francisco the "secret room" is Room 641A at 611 Folsom Street, the site of a large SBC phone building, three floors of which are occupied by AT&T. High-speed fiber-optic circuits come in on the 8th floor and run down to the 7th floor where they connect to routers for AT&T's WorldNet service, part of the latter's vital "Common Backbone." In order to snoop on these circuits, a special cabinet was installed and cabled to the "secret room" on the 6th floor to monitor the information going through the circuits. (The location code of the cabinet is 070177.04, which denotes the 7th floor, aisle 177 and bay 04.) The "secret room" itself is roughly 24-by-48 feet, containing perhaps a dozen cabinets including such equipment as Sun servers and two Juniper routers, plus an industrial-size air conditioner.

The normal work force of unionized technicians in the office are forbidden to enter the "secret room," which has a special combination lock on the main door. The telltale sign of an illicit government spy operation is the fact that only people with security clearance from the National Security Agency can enter this room. In practice this has meant that only one management-level technician works in there. Ironically, the one who set up the room was laid off in late 2003 in one of the company's endless "downsizings," but he was quickly replaced by another.

Plans for the "secret room" were fully drawn up by December 2002, curiously only four months after Darpa started awarding contracts for TIA. One 60-page document, identified as coming from "AT&T Labs Connectivity & Net Services" and authored by the labs' consultant Mathew F. Casamassima, is titled Study Group 3, LGX/Splitter Wiring, San Francisco and dated 12/10/02. This document addresses the special problem of trying to spy on fiber-optic circuits. Unlike copper wire circuits which emit electromagnetic fields that can be tapped into without disturbing the circuits, fiber-optic circuits do not "leak" their light signals. In order to monitor such communications, one has to physically cut into the fiber somehow and divert a portion of the light signal to see the information.

This problem is solved with "splitters" which literally split off a percentage of the light signal so it can be examined. This is the purpose of the special cabinet referred to above: Circuits are connected into it, the light signal is split into two signals, one of which is diverted to the "secret room." The cabinet is totally unnecessary for the circuit to perform -- in fact it introduces problems since the signal level is reduced by the splitter -- its only purpose is to enable a third party to examine the data flowing between sender and recipient on the internet.

The above-referenced document includes a diagram showing the splitting of the light signal, a portion of which is diverted to "SG3 Secure Room," i.e., the so-called "Study Group" spy room. Another page headlined "Cabinet Naming lists not only the "splitter" cabinet but also the equipment installed in the "SG3" room, including various Sun devices, and Juniper M40e and M160 "backbone" routers. PDF file 4 shows one of many tables detailing the connections between the "splitter" cabinet on the 7th floor (location 070177.04) and a cabinet in the "secret room" on the 6th floor (location 060903.01). Since the San Francisco "secret room" is numbered 3, the implication is that there are at least several more in other cities (Seattle, San Jose, Los Angeles and San Diego are some of the rumored locations), which likely are spread across the United States.

One of the devices in the "Cabinet Naming" list is particularly revealing as to the purpose of the "secret room": a Narus STA 6400. Narus is a 7-year-old company which, because of its particular niche, appeals not only to businessmen (it is backed by AT&T, JP Morgan and Intel, among others) but also to police, military and intelligence officials. Last November 13-14, for instance, Narus was the "Lead Sponsor" for a technical conference held in McLean, Virginia, titled "Intelligence Support Systems for Lawful Interception and Internet Surveillance." Police officials, FBI and DEA agents, and major telecommunications companies eager to cash in on the "war on terror" had gathered in the hometown of the CIA to discuss their special problems. Among the attendees were AT&T, BellSouth, MCI, Sprint and Verizon. Narus founder, Dr. Ori Cohen, gave a keynote speech. So what does the Narus STA 6400 do?

"The (Narus) STA Platform consists of standalone traffic analyzers that collect network and customer usage information in real time directly from the message.... These analyzers sit on the message pipe into the ISP (internet service provider) cloud rather than tap into each router or ISP device" (Telecommunications magazine, April 2000). A Narus press release (1 Dec., 1999) also boasts that its Semantic Traffic Analysis (STA) technology "captures comprehensive customer usage data ... and transforms it into actionable information.... (It) is the only technology that provides complete visibility for all internet applications."

To implement this scheme, WorldNet's high-speed data circuits already in service had to be rerouted to go through the special "splitter" cabinet. This was addressed in another document of 44 pages from AT&T Labs, titled SIMS, Splitter Cut-In and Test Procedure ,dated 01/13/03. "SIMS" is an unexplained reference to the secret room. Part of this reads as follows:

"A WMS (work) Ticket will be issued by the AT&T Bridgeton Network Operation Center (NOC) to charge time for performing the work described in this procedure document....
"This procedure covers the steps required to insert optical splitters into select live Common Backbone (CBB) OC3, OC12 and OC48 optical circuits."

The NOC referred to is in Bridgeton, Missouri, and controls WorldNet operations. (As a sign that government spying goes hand-in-hand with union-busting, the entire (Communication Workers of America) Local 6377 which had jurisdiction over the Bridgeton NOC was wiped out in early 2002 when AT&T fired the union work force and later rehired them as nonunion "management" employees.) The cut-in work was performed in 2003, and since then new circuits are connected through the "splitter" cabinet.

Another Cut-In and Test Proceduredocument dated January 24, 2003, provides diagrams of how AT&T Core Network circuits were to be run through the "splitter" cabinet. One page lists the circuit IDs of key Peering Links which were "cut-in" in February 2003, including ConXion, Verio, XO, Genuity, Qwest, PAIX, Allegiance, AboveNet, Global Crossing, C&W, UUNET, Level 3, Sprint, Telia, PSINet and Mae West. By the way, Mae West is one of two key internet nodal points in the United States (the other, Mae East, is in Vienna, Virginia). It's not just WorldNet customers who are being spied on -- it's the entire internet.

The next logical question is, what central command is collecting the data sent by the various "secret rooms"? One can only make educated guesses, but perhaps the answer was inadvertently given in the DOD Inspector General's report (cited above):

"For testing TIA capabilities, Darpa and the U.S. Army Intelligence and Security Command (INSCOM) created an operational research and development environment that uses real-time feedback. The main node of TIA is located at INSCOM (in Fort Belvoir, Virginia)."

Among the agencies participating or planning to participate in the INSCOM "testing" are the "National Security Agency, the Defense Intelligence Agency, the Central Intelligence Agency, the DOD Counterintelligence Field Activity, the U.S. Strategic Command, the Special Operations Command, the Joint Forces Command and the Joint Warfare Analysis Center." There are also "discussions" going on to bring in "non-DOD federal agencies" such as the FBI.

This is the infrastructure for an Orwellian police state. It must be shut down!


[/size]

[size="3"]3. AT&T titled Traffic Engineering for ISP Networks



This is the html version of the file http://www.cms.wisc.edu/~stochnet/abstract...ford_slides.ppt.
--------------------------------------------------------------------------------


Traffic Engineering for ISP Networks


Jennifer Rexford

Internet and Networking Systems

AT&T Labs - Research; Florham Park, NJ

http://www.research.att.com/~jrex


Joint work with Anja Feldmann, Albert Greenberg, Carsten Lund, Nick Reingold, and Fred True, and AT&T IP Services





Outline


Background
Internet architecture
Interdomain and intradomain routing
Internet service provider backbone
Traffic engineering
Optimizing network configuration to prevailing traffic
Requirements for topology, routing, and traffic info
Traffic demands
Volume of load between edges of the network
Measurement methodology
Analysis of the demands on AT&T’s IP Backbone




Internet Architecture


Divided into Autonomous Systems
Distinct regions of administrative control (~6000-7000)
Set of routers and links managed by a single institution
Service provider, company, university, …
Hierarchy of Autonomous Systems
Large, tier-1 provider with a nationwide backbone
Medium-sized regional provider with smaller backbone
Small network run by a single company or university
Interaction between Autonomous Systems
Internal topology is not shared between ASes
… but, neighboring ASes interact to coordinate routing


Autonomous Systems (ASes)


1 ,2, 3, 4, 5, 6, 7


Client


Web server


Path: 6, 5, 4, 3, 2, 1


Characteristics of the Internet


The Internet is
Decentralized (loose confederation of peers)
Self-configuring (no global registry of topology)
Stateless (limited information in the routers)
Connectionless (no fixed connection between hosts)
These attributes contribute
To the success of Internet
To the rapid growth of the Internet
… and the difficulty of controlling the Internet!



Internet – Interconnection of Networks


Internet Protocol
Transmit a single packet from one host to another
Packet includes the IP address of the sender and receiver
Packets may be lost, delayed, or delivered out of order
Hosts perform retransmission and reordering of packets
IP address
32-bit IP addresses divided into octets (12.34.158.5)
Allocated to institutions in contiguous blocks or prefixes
12.34.158.0/24 is a 24-bit prefix with 28 IP addresses
Routing of IP packets in the Internet is based on prefixes



Interdomain Routing (Between ASes)


ASes exchange info about who they can reach
Local policies for path selection (which to use?)
Local policies for route propagation (who to tell?)
Policies configured by the AS’s network operator



1, 2, 3


Client (12.34.158.5)


12.34.158.5


“I can reach 12.34.158.0/24”


“I can reach 12.34.158.0/24

via AS 1”



Internet Service Provider Backbone


modem banks,

business customers,

web/e-mail servers


neighboring providers


How should traffic be routed through the ISP backbone?



Intradomain Routing (Within an AS)


Routers exchange information to learn the topology
Routers determine “next hop” to reach other routers
Path selection based on link weights (shortest path)
Link weights configured by AS’s network operator
… to engineer the flow of traffic



3- 2- 2- 1 -1-3-1-4-5-3




Traffic Engineering in an ISP Backbone


Topology of the ISP backbone
Connectivity and capacity of routers and links
Traffic demands
Expected/offered load between points in the network
Routing configuration
Tunable rules for selecting a path for each traffic flow
Performance objective
Balanced load, low latency, service level agreements …
Question: Given the topology and traffic demands in an IP network, which routes should be used?




State-of-the-Art in IP Networks


Missing input information
The topology and traffic demands are often unknown
Traffic fluctuates over time (user behavior, new appls)
Topology changes over time (failures, growth, reconfig)
Primitive control over routing
The network does not adapt the routes to the load
The static routes are not optimized to the traffic
Routing parameters are changed manually by operators
(But, other than that, everything is under control…)





Example: Congested Link


Detecting that a link is congested
Utilization statistics reported every five minutes
Sample probe traffic suffers degraded performance
Customers complain (via the telephone network?)
Reasons why the link might be congested
Increase in demand between some set of src-dest pairs
Failed router/link within the AS causes routing change
Failure/reconfiguration in another AS changes routes
How to determine why the link is congested???
Need to know the cause, not just the manifestations!
How to alleviate the congestion on the link???




Link Utilization


Utilization: link color (high to low)


Requirements for Traffic Engineering


Models
Traffic demands
Network topology/configuration
Internet routing algorithms
Techniques for populating the models
Measuring/computing the traffic demands
Determining the network topology/configuration
Optimizing the routing parameters
Analysis of the traffic demands
Knowing how the demands fluctuates over time
Understanding the traffic engineering implications




Modeling Traffic Demands


Volume of traffic V(s,d,t)
From a particular source s
To a particular destination d
Over a particular time period t
Time period
Performance debugging -- minutes or tens of minutes
Time-of-day traffic engineering -- hours
Network design -- days to weeks
Sources and destinations
Individual hosts -- interesting, but huge!
Individual prefixes -- still big; not seen by any one AS!
Individual edge links in an ISP backbone -- hmmm….




Traffic Matrix


in


out


Traffic matrix: V(in,out,t) for all pairs (in,out)



Problem: Multiple Exit Points


ISP backbone is in the middle of the Internet
Multiple connections to other autonomous systems
Destination is reachable through multiple exit points
Selection of exit point depends on intradomain routes
Problem with traditional point-to-point models
Want to predict impact of changing intradomain routing
But, a change in routing may change the exit point!


1, 2, 3, 4



Traffic Demand


Definition: V(in, {out}, t)
Entry link (in)
Set of possible exit links ({out})
Time period (t)
Volume of traffic (V(in,{out},t))
Computing the traffic demands
Measure the traffic where it enters the ISP backbone
Identify the set of exit links where traffic could leave
Associate traffic with the entry link and set of exit links
Aggregate over all traffic with same in, {out}, and t




flow 1, flow 2, flow 3, flow 4


Measuring Flows Rather Than Packets


IP flow abstraction
Set of packets with “same” src and dest IP addresses
Packets that are “close” together in time (a few seconds)
The closest thing to a “call” in the Internet
Cisco NetFlow
Measure all flows between AT&T and neighbors
Extremely large amount of data (~100 GB/day)



NetFlow Data


Source and destination information
Source and destination IP addresses (hosts)
Source and destination port numbers (application)
Source and destination AS numbers
Routing information
Source and destination IP prefix (network address)
Input and output links at this router
Traffic information
Start and finish time of flow (in seconds)
Total number of bytes and packets in the flow




Identifying Where the Traffic Can Leave


Traffic flows
Each flow has a dest IP address (e.g., 12.34.156.5)
Each address belongs to a prefix (e.g., 12.34.156.0/24)
Forwarding tables
Each router has a table to forward a packet to “next hop”
Forwarding table maps a prefix to a “next hop” link
Process
Dump the forwarding table from each router
Identify the entries where the “next hop” is an exit link
Identify the set of exit links associated with each prefix
Associate a flow’s dest address with the set of exit links




Locating the Set of Exit Links for Prefix d


d , i, k

Prefix d: exit links {i, k}


Table entry: (d, i)


Table entry: (d, k)


Experimental Results: AT&T IP Backbone


Measurement for four days in November 1999
Netflow data
Forwarding tables
Topology and routing configuration
Traffic analysis
Distribution of traffic volume across demands
Small % of demands are responsible for most traffic
Time-of-day fluctuations in traffic volumes
U.S. business, U.S. residential, International
Stability of traffic demands across hours and days
Initial results suggest some stability, but need to study




Proportion of Traffic in Top Demands (Log Scale)


Time-of-Day Effects (San Francisco)

Traffic-Engineering Implications


Small number of demands contribute most traffic
Can optimize routes for just the heavy hitters
Can measuring a small fraction of the traffic
Must watch out for changes in load and set of exit links
Time-of-day fluctuations
Reoptimize routes a few times a day (three?)
Traffic (in)stability
Select routes that are good for different demand sets
Reoptimize routes after sudden changes in load




Traffic Flow Through AT&T’s IP Backbone


Color/size of node: proportional to traffic to this router (high to low)

Color/size of link: proportional to traffic carried (high to low)


Source node: public peering link (NAP) in New York
Destination nodes: AT&T access routers




Conclusions


Internet traffic engineering is hard
Decentralized (over 6000 Autonomous Systems)
Connectionless (traffic sent as individual packets)
Changing (topological changes, traffic fluctuations)
Traffic engineering requires knowing the demands
Interdomain traffic has multiple possible exit points
Demand as the load from entry to set of exit points
Not available from traditional measurement techniques
Measurement of traffic demands
Derivable from flow-level measurements at entry points
… and “next hop” forwarding info from exit points




Ongoing Work


Detailed analysis of traffic demands
Statistical properties (how to study stability?)
Implications for traffic engineering
Online computation of traffic demands
Distributed flow-measurement infrastructure
Online aggregation of flow data into demands
Network operations (“operations” research?)
Efficiently detecting sudden changes in traffic or routing
Optimizing routes based on topology and demands
Planning the design of the network over time
Getting the network to run itself…




Interesting Problems


Inferring the traffic demands from less information
sampling, active probes, inference from utilization
Optimizing routes subject to fluctuating demands
optimal routes per demand set vs. good for all sets
Techniques for analyzing stability of demand sets
multidimensional data (in, {out}, time)
Detecting shifts in the distribution of load
random changes vs. change in underlying distribution
Joint route optimization across multiple ASes
optimizing routes without divulging topology & traffic