IRS: So… Our Computer Crashed And Erased All Of Lois Lerner’s Emails

Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you:

No adverts like this in the forums anymore.
Times and dates in your local timezone.
Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance.
Email notifications for threads you want to watch closely.
Being a part of the oldest wargaming community on the net.

If you are already a member then feel free to login now.

**the shrouded lord**

I don't get it...

**dogma**

Grey Templar wrote:
If the IRS's systems worked as poorly as the ACA we'd definitely know about it.

No, we probably wouldn't. "The IRS has a bad email server!" isn't really a sexy headline.

**Grey Templar**

Do you really think they have a server dedicated for their e-mail? And only it had a problem? And the e-mails which happened to disappear were ones which were being demanded in an investigation. And if said e-mails had actually disappeared when they claim they did then we would not have heard about it immediately upon those e-mails being wanted?

Probability is spilling over into a mathematical certainty that they were deliberately destroyed.

**whembly**

Ouze wrote:
Grey Templar wrote:
If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

Although I, like Whembly and the guy from the article, am not privy to the IRS's email system, I am nonetheless quite confident in speculating it's absolutely terrible software, contracted out the lowest bidder who also was connected to someone in the appropriations process, that the specs changed constant, the program manager changed things during development without documenting anything, and it cost at least 3 times what it was supposed to while delivering 30% of the features it was sold upon.

They use Microsoft Exchange Servers (Outlook).

If you do a "mx lookup" of irs.gov... that'll show at least 10 external facing mx records.

That's a big system... HUGE.

Systems that big are replicated... that are robust and fault tolerant.

I have been in the IT industry long enough to deal with Exchange Server in various capacity. For instance, the IRS is most likely using Exchange 2010 or newer, which has built in features for litigation hold, retention policies, and database availability groups which allow you to copy mailbox databases to multiple servers.

Couple that with storage array replication to other sites, backup to disk and/or tape, its damn near impossible to lose data by accident. You have to be purposefully malicious to somehow have email lost and apparently, the IRS believes it would be easy to get away with this excuse with the general public because they don’t understand how such systems is architected.

One of the biggest irony is that private businesses have to follow email retention policies depending on what industry they’re in. The Sarb-Ox compliance requires email retention (min of 7 years I believe) or you can face fines. MS Exchange would have these functionalities inherently. Somehow, the IRS is saying that our federal government can’t be bothered to ensure data is protected…yet they want to run other aspects of our lives(namely the PPACA).... ???

O.o

Congress should not only grill whomever is in charge of messaging at the IRS and their IT director/CIO, but anyone who’s this incompetent should get the hell out of my industry. That's how embarrassing this is...

Here's a list of questions Congress should ask for each of the IT executives:
a) Given the redundant infrastructure, disaster recovery and business continuity facilities, and other preparations designed to maintain the IRS's technical operations, describe how the agency could state that a small fraction of the email related to Lois Lerner could be reported as "destroyed".
b) Describe the logical architecture of backups for the Exchange system including DR sites, offline storage of backup tapes, etc.
c) Describe the interval of full and incremental backups for the Exchange system.
d) Describe a scenario in which all backups from Lerner for the years in question could be destroyed.

This isn't hard guys. Very basic IT industry concepts.

**SagesStone**

Clearly they have about 10 crap servers then right?

I don't even think a medium sized company could get away with this crap. If they've had a crash and actually lost important stuff like this, in an organisation specifically made to store records and such, then obviously this would call for a pretty big overhaul of their system to make sure it doesn't happen again. Then whoever was in charge of it would have a bit to answer for.

This is like some guy building a house, he has just finished the roof looks around then goes "OH FETH ME! I FORGOT TO PUT THE FLOORS IN!". This is like the tech support asking you to shut down your computer with a bathtub to stop you wearing out the power button. Or like having a new traffic signal design; laying on its side with all the lights green and where they blink every .5 secs it means stop every .6 times it means go and every .4 secs it means bridge out.

**motyak**

Has anyone acknowledged d-usa's point that this is pretty much a recurring theme? If it happens with the next admin, then it's happened R-D-R then would we stop getting up in arms about it while pretending the other side didn't do it?

**d-usa**

motyak wrote:
Has anyone acknowledged d-usa's point that this is pretty much a recurring theme? If it happens with the next admin, then it's happened R-D-R then would we stop getting up in arms about it while pretending the other side didn't do it?

Just to be clear though, I don't think it is right simply because both sides are doing it. I do think it's complete bs and an insult to all of us to pretend this really happened. I don't think we should care less about it happening simply because of who the guys in charge are or if we disagree with them.

I'm just pragmatic enough to realize that neither party has any vested interest in actually fixing the problem. So other than grand-standing nothing will happen because they want to get away with it later as well.

**motyak**

Oh no, I'm not saying it's a good thing either. I was just wondering what peoples responses to your point are.

**dogma**

Grey Templar wrote:
Do you really think they have a server dedicated for their e-mail?

Yes. It is a massive organization which processes thousands of emails every day. That is a huge amount of data, which would require a dedicated server.

Grey Templar wrote:

And if said e-mails had actually disappeared when they claim they did then we would not have heard about it immediately upon those e-mails being wanted?

I was unaware that the IRS made a specific claim regarding the date of the disappearance of the emails. But at any rate, no, we would not.

Missing data can only be accounted for by way of the presence of other data. Analogically the sequence (1,2,3,5) only appears to be missing 4 because (1,2,3,5) establishes a consistent pattern with which we are all familiar. But if we're discussing something more complicated, like the email habits of an IRS director, patterns are only forthcoming after a good deal of research; and even then specifics will not be available.

**d-usa**

We also wouldn't know that "just those emails" are missing.

There is no way of knowing what, if any other, emails were wiped out unless you have prior knowledge of what ever other email stored in the same area would be.

**Dreadclaw69**

http://politicalticker.blogs.cnn.com/2014/06/14/irs-loses-subpoenaed-lerner-e-mails-in-computer-crash/?hpt=hp_t2

(CNN) – Republican lawmakers are slamming the Internal Revenue Service, with one calling for an immediate investigation, after the agency notified Congress Friday that it was unable to recover former official Lois Lerner’s e-mails from January 2009 to April 2011 because of a computer crash.

The agency made the disclosure in a letter sent to Congressional investigators Friday afternoon, according a statement from Rep. Charles Boustany Jr., R-Louisiana.

Lerner, former director of the IRS’ exempt organizations division, is a central figure in the IRS controversy involving the agency’s targeting of tea party and other interest-specific groups. Her e-mails were subpoenaed by House Oversight Committee Chairman Darrell Issa, R-California, in an attempt to learn whether other outside agencies were potentially involved in the flagging of applications from those groups.

In a statement, the IRS said that Lerner tried to recover her e-mails after her hard drive broke down but was unable to do so. The agency said it then worked backward to recover the missing e-mails from recipients of the messages.

"At the time, Ms. Lerner asked IRS IT professionals to restore her hard drive, but they were unable to do so,” the IRS said. “Nonetheless, the IRS has or will produce 24,000 Lerner e-mails from this 2009-2011 time period, largely from the files of the other 82 individuals. The IRS's production to Congress of the 67,000 Lerner e-mails is nearly complete."

On Friday, House Ways and Means Committee Chairman Dave Camp issued a statement questioning the credibility of the IRS’ compliance with the congressional investigation for not having raised the issue sooner. He also called for an immediate investigation.

“The fact that I am just learning about this, over a year into the investigation, is completely unacceptable and now calls into question the credibility of the IRS’s response to Congressional inquiries,” Camp said. “There needs to be an immediate investigation and forensic audit by Department of Justice as well as the Inspector General.

But the IRS said it advised the Senate Finance Committee three months ago that it had completed the production of materials related to the committee’s investigation and that the missing Lerner e-mails are part of an additional search.

Issa went a step further, saying that the crash was “convenient” for the Obama administration, and that they are playing games. Echoing Camp’s comments on credibility, Issa said that the supposed loss of Lerner’s documents damages the credibility of claims that the IRS is complying with congressional requests to seek the truth.

Senate Finance Committee Ranking Member Orrin Hatch, R-Utah, said that while he is concerned about the missing documents, he is also “greatly troubled” by the administration’s failure to notify his committee of the computer crash when they first became aware of it. This new information, he says, “will without question delay our investigation.”

Hatch says he is looking forward to a “forthright discussion” with IRS Commissioner John Koskinen on the matter next week.

The IRS insists that it has made unprecedented efforts to produce the documents needed to help complete the investigations. According to a statement, their efforts to respond to Congress’ requests have involved more than 250 IRS employees working more than 120,000 hours at a direct cost of nearly $10 million. The agency says it has thus far delivered more than 750,000 pages of documents to Congress.

Senate Finance Committee Chairman Ron Wyden, D-Oregon, was less critical, but said his committee is working to determine if any documents were withheld. “We’re working to finalize the Committee’s bipartisan investigation of the IRS’s review and processing of applications for tax exempt status,” Wyden said in a statement. “To date, our effort has included a review of hundreds of thousands of related pages of documents and over 30 interviews. We are now working with the IRS to determine if any relevant documents were not provided and will continue to work toward a release of our bipartisan report.”

Yeah.... that is the equivalent of saying that the dog ate her homework

**-Shrike-**

So how long ago did her computer "crash"?

**d-usa**

Seems relevant...

**Ouze**

whembly wrote:
I have been in the IT industry long enough to deal with Exchange Server in various capacity. For instance, the IRS is most likely using (snip)

Again, neither you nor anyone else in this thread knows feth-all about the specifics of the IT infrastructure of the IRS. It's all criticism based upon speculation. You're assuming they use Exchange, you are assuming what version, you're assuming offsite backup, and you're assuming tape. All assumptions predicated on the fact you see malice where incompetence adequately answers the question. I don't even know if SOX applies to the IRS.

**MrDwhitey**

Not knowing anything can't stop us from typing furiously about things we want to hate, Ouze!

**Ouze**

I hope the fact that I pointed at the Bush administration and speculated that intentional incompetence is responsible, rather than a Machiavellian scheme, doesn't make it seem that I excuse what happened. I think there needs to be a congressional subcommittee and make sure there was no violation of the law, and that if anyone did break the law, they are prosecuted. I'm not saying it's OK, just saying this is what usually happens and nothing is typically the result. Still, we need to get this Kabuki going.

Automatically Appended Next Post:
Here is an article that does finally have some specifics.

In part,

Now, the Michigan Republican says the Obama administration should conduct an “administration-wide search and production of any emails to or from Lois Lerner.”

The IRS explains in the letter that it has not always backed up all employee emails due to the cost the agency would incur for allowing 90,000 employees to store their information on the IRS’s internal system.

Currently, IRS employees have the capacity to store about 6,000 emails in their active Outlook email boxes, which are saved on the IRS centralized network. But the letter and background document sent to the Hill Friday said they could only store about 1,800 emails in their active folders prior to July 2011.

When their inboxes were full, IRS employees had to make room by either deleting emails or archiving them on their personal computers. Archived data were not stored by the IRS but by the individual.
Such archived emails on Lerner’s computer were what were lost when her computer crashed.

“Any of Ms. Lerner’s email that was only stored on that computer’s hard drive would have been lost when the hard drive crashed and could not be recovered,” the letter reads.

Overall, more than 250 IRS employees have spent more than 120,000 hours digging up documents and emails for congressional investigators, spending $10 million.

Sen. Ron Wyden (D-Ore.), chairman of the Finance Committee that is also probing the matter, released a more neutral statement.

“We’re working to finalize the committee’s bipartisan investigation of the IRS’s review and processing of applications for tax-exempt status,” Wyden said. “To date, our effort has included a review of hundreds of thousands of related pages of documents and over 30 interviews. We are now working with the IRS to determine if any relevant documents were not provided and will continue to work toward a release of our bipartisan report.”

**Dreadclaw69**

Ouze wrote:
whembly wrote:
I have been in the IT industry long enough to deal with Exchange Server in various capacity. For instance, the IRS is most likely using (snip)

Again, neither you nor anyone else in this thread knows feth-all about the specifics of the IT infrastructure of the IRS. It's all criticism based upon speculation. You're assuming they use Exchange, you are assuming what version, you're assuming offsite backup, and you're assuming tape. All assumptions predicated on the fact you see malice where incompetence adequately answers the question. I don't even know if SOX applies to the IRS.

We may not know the specifics of the infrastructure, but given the IRs's excuse we don't need to;
http://politicalticker.blogs.cnn.com/2014/06/14/irs-loses-subpoenaed-lerner-e-mails-in-computer-crash/?hpt=hp_t2

(CNN) –
In a statement, the IRS said that Lerner tried to recover her e-mails after her hard drive broke down but was unable to do so. The agency said it then worked backward to recover the missing e-mails from recipients of the messages.

"At the time, Ms. Lerner asked IRS IT professionals to restore her hard drive, but they were unable to do so,” the IRS said. “Nonetheless, the IRS has or will produce 24,000 Lerner e-mails from this 2009-2011 time period, largely from the files of the other 82 individuals. The IRS's production to Congress of the 67,000 Lerner e-mails is nearly complete."

To whit, you don't lose your emails if your computer's hard drive dies. This excuse is so flimsy it can be spat through

**whembly**

Ouze wrote:
whembly wrote:
I have been in the IT industry long enough to deal with Exchange Server in various capacity. For instance, the IRS is most likely using (snip)

Again, neither you nor anyone else in this thread knows feth-all about the specifics of the IT infrastructure of the IRS. It's all criticism based upon speculation. You're assuming they use Exchange, you are assuming what version, you're assuming offsite backup, and you're assuming tape. All assumptions predicated on the fact you see malice where incompetence adequately answers the question. I don't even know if SOX applies to the IRS.

The IRS does use exchange.

They even post their email usage standard online:
http://www.irs.gov/irm/part1/irm_01-010-003.html

But, you're right, SOX doesn't apply to any government agency as far as I can see. Interesting, eh?

**SagesStone**

(CNN) –
In a statement, the IRS said that Lerner tried to recover her e-mails after her hard drive broke down but was unable to do so. The agency said it then worked backward to recover the missing e-mails from recipients of the messages.

"At the time, Ms. Lerner asked IRS IT professionals to restore her hard drive, but they were unable to do so,” the IRS said. “Nonetheless, the IRS has or will produce 24,000 Lerner e-mails from this 2009-2011 time period, largely from the files of the other 82 individuals. The IRS's production to Congress of the 67,000 Lerner e-mails is nearly complete."

IRS IT professionals hard at work to recover the lost emails.

**dogma**

Dreadclaw69 wrote:

To whit, you don't lose your emails if your computer's hard drive dies. This excuse is so flimsy it can be spat through

If they're exclusively stored on the drive which failed you very well might.

Additionally, I'm uncertain as to what your quote was meant to prove; given that it refers to emails produced from the records of people that are not Lois Lerner.

whembly wrote:

The IRS does use exchange.

They even post their email usage standard online:
http://www.irs.gov/irm/part1/irm_01-010-003.html

What part of that indicates that the IRS uses Exchange?

**Avatar 720**

dogma wrote:
Dreadclaw69 wrote:

To whit, you don't lose your emails if your computer's hard drive dies. This excuse is so flimsy it can be spat through

If they're exclusively stored on the drive which failed you very well might.

Additionally, I'm uncertain as to what your quote was meant to prove; given that it refers to emails produced from the records of people that are not Lois Lerner.

whembly wrote:

The IRS does use exchange.

They even post their email usage standard online:
http://www.irs.gov/irm/part1/irm_01-010-003.html

What part of that indicates that the IRS uses Exchange?

As someone untrained in deciphering that link I found two references to Exchange, implying that the IRS uses it in at least some capacity.

**-Shrike-**

I agree with Avatar 720 here. Perhaps this means something else to someone who does this for a living,but I'd certainly say they use Exchange.

Refrain from sending large attachments to work groups or audiences. Remember every email message and any attachments, embedded graphics and photographs require a copy for each Exchange server store where each recipient’s mailbox resides. Instead store the document on an IRS public web archive or SharePoint repository and insert a hyperlink into the message. Ensure the permissions allow access by all recipients prior to sending the message.

**Dreadclaw69**

Avatar 720 wrote:
As someone untrained in deciphering that link I found two references to Exchange, implying that the IRS uses it in at least some capacity.

You are correct, the IRS uses Exchange and have admitted that fact. Also it is highly improbable (I would say virtually impossible) that the emails were solely stored on the HDD of Lerner's computer.

**d-usa**

Dreadclaw69 wrote:
Avatar 720 wrote:
As someone untrained in deciphering that link I found two references to Exchange, implying that the IRS uses it in at least some capacity.

You are correct, the IRS uses Exchange and have admitted that fact. Also it is highly improbable (I would say virtually impossible) that the emails were solely stored on the HDD of Lerner's computer.

I can actually somewhat believe that.

We have limited Outlook mailboxes as well. And once full we have to delete them to make room on the servers. They don't have 6 years worth of "yeah, I'll work overtime" and "that doctor was mean to me" emails that I wrote, but then again I'm not exactly anyone important either though.

**dogma**

-Shrike- wrote:
I agree with Avatar 720 here. Perhaps this means something else to someone who does this for a living,but I'd certainly say they use Exchange.

Yeah, I missed that. That's what I get for using a document search instead of reading carefully.

Dreadclaw69 wrote:

You are correct, the IRS uses Exchange and have admitted that fact. Also it is highly improbable (I would say virtually impossible) that the emails were solely stored on the HDD of Lerner's computer.

According to the article Ouze posted at the time in question IRS personnel could only store 1800 emails on the Exchange server, a number I suspect Ms. Lerner exceeded several times over.

**LoneLictor**

As time goes on, the less I care about political parties and the more I care about individual politicians. Donkeys and elephants might be whole different species, but when they gak it smells the same.

**Frazzled**

Ouze wrote:
whembly wrote:
I have been in the IT industry long enough to deal with Exchange Server in various capacity. For instance, the IRS is most likely using (snip)

Again, neither you nor anyone else in this thread knows feth-all about the specifics of the IT infrastructure of the IRS. It's all criticism based upon speculation. You're assuming they use Exchange, you are assuming what version, you're assuming offsite backup, and you're assuming tape. All assumptions predicated on the fact you see malice where incompetence adequately answers the question. I don't even know if SOX applies to the IRS.

Dude I'm not assuming they've even made it beyond punchcards...

**Dreadclaw69**

d-usa wrote:
I can actually somewhat believe that.

We have limited Outlook mailboxes as well. And once full we have to delete them to make room on the servers. They don't have 6 years worth of "yeah, I'll work overtime" and "that doctor was mean to me" emails that I wrote, but then again I'm not exactly anyone important either though.

For the rank and file I can believe that there is limited inbox space. I agree that as inboxs fill up that you make a decision to delete based on what is mundane v what is important. At the time this was still a ongoing issue within the IRS, it stands to reason that any discussion about an ongoing process (especially one that is legitimate) would not be removed by the parties involved. It defies any logic.

What I cannot believe is that by deleting an email on a sender's computer that it is magically deleted on the recipient's also.

**Ouze**

Dreadclaw69 wrote:
What I cannot believe is that by deleting an email on a sender's computer that it is magically deleted on the recipient's also.

Not exactly, but if the sender deletes it, and the recipient also deletes or archives it (locally), the server copy is removed, if it's set up that way.

In other words, unless there is a retention policy to prevent this, if I send you an email, and then I delete it from my sent items (or have it set to archive locally), and you get it, decide it's not important, and delete it, on day 31 it's gone, essentially. There is a single copy on my local computer hard drive, assuming it's archived there, and the copy you deleted is generally removed after sitting in deleted items for 30 days.

When I was supporting several different companies, at least one was set up this way - a big, fortune 1000 mining company. They had migrated from Groupwise and had no idea how to administer exchange, and had a terrible IT admin in general. I have no idea what they would do if they were sued and forced to produce email for litigation.

Another company I formerly supported, a 6 billion dollar realty company, tried to avoid this by having all of the PST's sorted on a share drive. The archives were routinely allowed to grow far, far in excess of the 2gb break size upon which corrupted data moves from being a possibility to a an inevitability. Yes, they had redundant backups and tape backups, as many copies of the unusable, corrupted data as you'd like.

Spoiler:

The point of this story is that, in my experience in the IT field, you'd be amazed at some of the incompetence and unreliability you find in real life versus how you expect things to work. Many companies resent that they have to spend money on IT at all and continuously underfund their departments, which results in expensive losses in productivity. Penny wise, pound foolish is the order of the day. I've been doing this almost 20 years, and I've really only worked for two companies that actually had intelligent IT planning, and one of which is because due to their business, they were often attacked by external forces of phishers hackers, and miscreants - they recognized early on that cheap IT is super expensive when you have company secrets that are worth something.

**Co'tor Shas**

I wonder how the IRS's system is?