IRS: So… Our Computer Crashed And Erased All Of Lois Lerner’s Emails

Forum adverts like this one are shown to any user who is not logged in. Join us by filling out a tiny 3 field form and you will get your own, free, dakka user account which gives a good range of benefits to you:

No adverts like this in the forums anymore.
Times and dates in your local timezone.
Full tracking of what you have read so you can skip to your first unread post, easily see what has changed since you last logged in, and easily see what is new at a glance.
Email notifications for threads you want to watch closely.
Being a part of the oldest wargaming community on the net.

If you are already a member then feel free to login now.

**whembly**

This is probably one of the funniest things I've heard - EVER.

A "computer crash" wiped out years of a single individuals emails. That is BS of the highest order...

An mx lookup of irs.gov reveals 10 external facing mx records.

That's a big system, big systems are replicated, replicated systems are robust and fault tolerant.

In other words it's complete and utter BS.

The fact that anyone in the IRS is suggesting this is utterly preposterous.

Lerner taking the 5th, one can infer that she has something to hide. It was my belief that she orchastrated it illegally and wants to thwart the investigation to protect herself.

THIS tells me that it's much bigger than that... high level House/Senate/Presidential level.

**Jihadin**

I cannot fathom having to finance multiple types of secure internet "channels" while there is one in heavy use. I mean with just one type already in place being military presence in areas, I am unsure that one "cluster" can be used to service all government entities. I guess how they would do it is each entity would have their version of a cluster concerning their area and use maybe a military satellite or something to communicate outside of their cluster if long distance or maybe a individually secure "wire" within maybe city limits branching out to others. I am guessing all of this for I am unsure how they would do it

**Frazzled**

Ahtman wrote:
Jihadin wrote:
Gentleman_Jellyfish wrote:
Jihadin wrote:
Deleted

Gotta watch out for the NSA

Or whoever it is

CYA

YMCA?

You rang?

**whembly**

From Sharyl Attkisson's site:
Lois Lerner’s Lost Emails: Questions for the IRS

The news came late in the day on Friday the 13th.

According to the House Ways and Means Committee, the IRS reports having “lost” former IRS manager Lois Lerner’s emails to and from other IRS employees sent between January of 2009 and April of 2011 due to a ‘computer crash.’

In light of the disclosure, these are some of the logical requests that should be made of the IRS:

-Please provide a timeline of the crash and documentation covering when it was first discovered and by whom; when, how and by whom it was learned that materials were lost; the official documentation reporting the crash and federal data loss; documentation reflecting all attempts to recover the materials; and the remediation records documenting the fix. This material should include the names of all officials and technicians involved, as well as all internal communications about the matter.
-Please provide all documents and emails that refer to the crash from the time that it happened through the IRS’ disclosure to Congress Friday that it had occurred.
-Please provide the documents that show the computer crash and lost data were appropriately reported to the required entities including any contractor servicing the IRS. If the incident was not reported, please explain why.
-Please provide a list summarizing what other data was irretrievably lost in the computer crash. If the loss involved any personal data, was the loss disclosed to those impacted? If not, why?
-Please provide documentation reflecting any security analyses done to assess the impact of the crash and lost materials. If such analyses were not performed, why not?
-Please provide documentation showing the steps taken to recover the material, and the names of all technicians who attempted the recovery.
-Please explain why redundancies required for federal systems were either not used or were not effective in restoring the lost materials, and provide documentation showing how this shortfall has been remediated.
-Please provide any documents reflecting an investigation into how the crash resulted in the irretrievable loss of federal data and what factors were found to be responsible for the existence of this situation.
-I would also ask for those who discovered and reported the crash to testify under oath, as well as any officials who reported the materials as having been irretrievably lost.
The Committee had requested the Lerner emails as part of its investigation into to the targeting of conservative non-profits by the IRS. The Obama administration has denied any corruption or intentional wrongdoing. Lerner took the Fifth when asked to testify to Congress. The House of Representatives subsequently held her in contempt. The lost materials are said to include any communications that may have occurred between Lerner and outside agencies or groups such as the White House, the Treasury Department, the Department of Justice, the Federal Elections Commission and the offices of Democrats.

House and Ways Committee Chairman Dave Camp (R-Mich.) says that along with providing news of the emails that have been lost, the IRS suggested in the same letter to Congress that it end its investigation.

The late disclosure of the lost emails may be reason to disregard the suggestion.

As Issa asked recently:

“If there wasn’t nefarious conduct that went much higher than Lois Lerner in the IRS targeting scandal, why are they playing these games?” asked Issa.

Why indeed?

**hotsauceman1**

Did it erase my tax forms aswell?
I mean, A computer that selectively erases when it crashes is amazing tech, What is the technology IRS has that we.

**whembly**

hotsauceman1 wrote:
Did it erase my tax forms aswell?
I mean, A computer that selectively erases when it crashes is amazing tech, What is the technology IRS has that we.

Don't worry... the NSA may have it.

Looks like Representative Stockman already asked the NSA.

Stockman asks NSA for Lois Lerner metadata after IRS claims ‘glitch’ erased all incriminating emails

‘Barack Obama has brought us Jimmy Carter’s economy and Richard Nixon’s excuses’
WASHINGTON – Congressman Steve Stockman Friday asked the National Security Agency to turn over all its metadata on the email accounts of former Internal Revenue Service Exempt Organizations division director Lois Lerner for the period between January 2009 and April 2011.

The request comes just hours after the IRS claimed it “lost” all of Lerner’s emails to or from Lerner and outside agencies or groups during that period, in which she allegedly coordinated with the White House, House Democrats and political groups to harass and deny tax-exempt status to groups critical of the President. The IRS blames a “computer glitch” for erasing the emails which could have implicated Agency employees in illegal activity.

“I have asked NSA Director Rogers to send me all metadata his agency has collected on Lois Lerner’s email accounts for the period which the House sought records,” said Stockman. “The metadata will establish who Lerner contacted and when, which helps investigators determine the extent of illegal activity by the IRS.”

“The claim incriminating communications were erased by a glitch conjures memories of Rose Mary Woods,” said Stockman. “Barack Obama has brought us Jimmy Carter’s economy and Richard Nixon’s excuses.”

The text of the letter follows:

June 13, 2014

Admiral Michael S. Rogers
Director, National Security Agency
Fort Meade, MD 20755

Admiral Rogers:

First, thank you for your 33 years of, and continued service to, our country.

Second, as you probably read, the Internal Revenue Service informed the House Ways and Means Committee today they claim to “lost” all emails from former Exempt Organizations division director Lois Lerner for the period between January 2009 and April 2011.

According to chairman Camp, “The IRS claims it cannot produce emails written only to or from Lerner and outside agencies or groups, such as the White House, Treasury, Department of Justice, FEC, or Democrat offices” due to a “computer glitch.”

I am writing to request the Agency produce all metadata it has collected on all of Ms. Lerner’s email accounts for the period between January 2009 and April 2011.

The data may be transmitted to our Communications Director at Donny@mail.house.gov.

Your prompt cooperation in this matter will be greatly appreciated and will help establish how IRS and other personnel violated rights protected by the First Amendment.

Warmest wishes,

STEVE STOCKMAN
Member of Congress

**Grey Templar**

I wonder if whoever was responsible can be charged with obstruction of justice? And contempt of Congress.

**d-usa**

Grey Templar wrote:
I wonder if whoever was responsible can be charged with obstruction of justice? And contempt of Congress.

Probably not. My guess is that we can find a metric crap-ton of circumstantial evidence that this is a line of crap that, combined with common sense, make it clear that it's all just an attempt to get out of providing these emails.

But I am going to guess that there are just enough holes that if you try to prove that it was all deliberate you would never get past reasonable doubt that it could have actually happened.

We all know what happened there. But unless Congress has iron-clad proof that somebody deleted all those emails and then said "oops, system crash" there will never be any charges.

Besides, next time around the other guys want to be able to bury evidence themselves.

**Grey Templar**

From what I know of computers, its very difficult to totally erase things without actually destroying the actual memory chips.

Its also fairly trivial to recover lost data as long as the memory chips are intact and new data hasn't been rewritten over it(deleting things just deletes the directory, the data is still there till it gets overwritten)

I doubt the IRS guys destroyed their computers.

I'll bet they'll say all the backups crashed as well.

**whembly**

Put the IT folks under oath.

Does everyone know that the IRS cannot lose emails, right? These email servers are inherently clustered against failure, and that the clusters are also redundant onto themselves, and that the entire email database is not only backed up daily, it is backed up, continuously. Not only are the email servers clustered against failure, but the Storage Arrays - SAN, (which are completely separate from the email application servers) are also fail over clustered and redundant. In addition the backup tapes are stored offsite, generally by secured vendor.

So, even if the email servers all crashed at once (an occurrence with a likelihood of less than .001% (five 9s), or less than 4 seconds of outage in any given year), everything ever written to the storage arrays are still on the storage arrays and cannot be deleted without basically turning all the storage into a smoking pile of radioactive dust.

We are being lied to. Again. By the same people.

Congress can subpoena anyone. Subpoena the IT staff workers in charge of Email and the other wholly separate group that manages the IT Storage infrastructure. And ask them to produce the email or risk jail.

Even the FBI is known for their ability to recover data that were forcibly destroyed. User their expertise.

Do that and the emails will be on their desk by morning.

Automatically Appended Next Post:

Grey Templar wrote:
From what I know of computers, its very difficult to totally erase things without actually destroying the actual memory chips.

Its also fairly trivial to recover lost data as long as the memory chips are intact and new data hasn't been rewritten over it(deleting things just deletes the directory, the data is still there till it gets overwritten)

I doubt the IRS guys destroyed their computers.

I'll bet they'll say all the backups crashed as well.

Data is written across storage arrays, not "memory chips".

Here's a decent story:
Veteran IT Professional Gives Six Reasons Why the IRS’ Claim That It ‘Lost’ Two Years of Lois Lerner’s Emails Is ‘Simply Not Feasible’
A veteran IT professional tells TheBlaze that the IRS’ claim that the agency lost two years’ worth of former IRS official Lois Lerner’s emails is “simply not feasible.”

On Friday, members of Congress revealed that the IRS would not be able to hand over Lerner’s emails to and from other IRS employees from January 2009 to April 2011, possibly due to a “glitch” or “crash.” Lawmakers were seeking the emails as part of their investigation into the IRS targeting scandal.

Norman Cillo, an Army veteran who worked in intelligence and a former program manager at Microsoft, argued it is very difficult to lose emails for good and laid out six reasons why he believes Congress is “being lied to” about the Lerner emails:

1. I believe the government uses Microsoft Exchange for their email servers. They have built-in exchange mail database redundancy. So, unless they did not follow Microsofts recommendations they are telling a falsehood. You can see by the diagram below that if you have three servers in a DAG you have three copies of the database.

2. Every IT organization that I know of has hotswappable disk drives. Every server built since 2000 has them. Meaning that if a single disk goes bad it’s easy to replace.

3. ALL Servers use some form of RAID technology. The only way that data can be totally lost (Meaning difficult to bring back) is if more than a single disk goes before the first bad disk is replaced. In the diagram below you can see that its possible to lose a single disk and still keep the data.

4. If the server crashed (Hardware failure other than disks), then the disks that contain the DATA for the Exchange database is still available because the server hardware and disks are exchangeable. Meaning that if I have another server with the same hardware in it, I can put the disks in and everything should boot right up.

5. All email servers in a professional organization use TAPE backup. Meaning if all the above fails, you can restore the server using the TAPE backups.

6. If they are talking about her local PC, then it’s a simple matter of going to the servers which have the email and getting them from the servers. If the servers have removed the data you can still get them by using the backups of the servers to recover the emails.

However, Cillo, who has been working in IT for roughly 16 years and is currently a consultant for a tech company, said it’s possible the IRS is telling the truth if the federal agency is “totally mismanaged and has the worst IT department ever.”

Other than that, it’s just not “feasible,” he told TheBlaze. “If the IRS’ email server is in such a state that they only have one copy of data and the server crashes and it’s gone, I’ve never heard of such a thing.”

“I don’t know of any email administrator that doesn’t have at least three ways of getting that mail back,” he added. “It’s either on the disks or it’s on a TAPE backup someplace or in an archive server. There are at least three ways the government can get those emails.”

**Grey Templar**

So can the insanely low probability of the incident occurring be used to prove beyond reasonable doubt that it didn't happen?

**whembly**

Grey Templar wrote:
So can the insanely low probability of the incident occurring be used to prove beyond reasonable doubt that it didn't happen?

Yes, that's what we (IT professionals) are trying to say.

**d-usa**

whembly wrote:
Put the IT folks under oath.

And?

"We didn't delete them on purpose, the system crashed."
"Thanks, you may step down."

Now what?

Does everyone know that the IRS cannot lose emails, right? These email servers are inherently clustered against failure, and that the clusters are also redundant onto themselves, and that the entire email database is not only backed up daily, it is backed up, continuously. Not only are the email servers clustered against failure, but the Storage Arrays - SAN, (which are completely separate from the email application servers) are also fail over clustered and redundant. In addition the backup tapes are stored offsite, generally by secured vendor.

So, even if the email servers all crashed at once (an occurrence with a likelihood of less than .001% (five 9s), or less than 4 seconds of outage in any given year), everything ever written to the storage arrays are still on the storage arrays and cannot be deleted without basically turning all the storage into a smoking pile of radioactive dust.

We are being lied to. Again. By the same people.

You are never going to be able to prove that.

We all know it. But it won't hold up in court.

Congress can subpoena anyone. Subpoena the IT staff workers in charge of Email and the other wholly separate group that manages the IT Storage infrastructure. And ask them to produce the email or risk jail.

That's just stupid.

You can't throw somebody in jail unless you can prove that they lied. Which is not going to happen.

Republicans are okay with that. They are going to scream and put on a dog and pony show for everyone to fake their outrage, but they want to be able to do the same down the line.

Even the FBI is known for their ability to recover data that were forcibly destroyed. User their expertise.

Do that and the emails will be on their desk by morning.

They are not going to do that because they have zero reason for wanting to take away the ability to do the same. We don't have proof that a crime occured to begin with other than "We are all pretty damn sure that this story is bs." The FBI can't just go digging aorund in other agencies without any sort of court order.

You start with higher ups in the IRS looking into things, then the OIG, then some more hearings, then MAYBE the OIG calls the FBI to ask for help, then some more hearings, then some other scandal that is new and juicy comes along and/or a Republican is in the White House and the whole thing gets dropped because neither party wants this to change.

**whembly**

d-usa wrote:
whembly wrote:
Put the IT folks under oath.

And?

"We didn't delete them on purpose, the system crashed."
"Thanks, you may step down."

Now what?

Do you even know what that means? They're lying.

That's like saying their name is Brad Pitt.

Does everyone know that the IRS cannot lose emails, right? These email servers are inherently clustered against failure, and that the clusters are also redundant onto themselves, and that the entire email database is not only backed up daily, it is backed up, continuously. Not only are the email servers clustered against failure, but the Storage Arrays - SAN, (which are completely separate from the email application servers) are also fail over clustered and redundant. In addition the backup tapes are stored offsite, generally by secured vendor.

So, even if the email servers all crashed at once (an occurrence with a likelihood of less than .001% (five 9s), or less than 4 seconds of outage in any given year), everything ever written to the storage arrays are still on the storage arrays and cannot be deleted without basically turning all the storage into a smoking pile of radioactive dust.

We are being lied to. Again. By the same people.

You are never going to be able to prove that.

We all know it. But it won't hold up in court.

We can prove it. Determine how the email system is architected. Really... it's as simple as that.

Congress can subpoena anyone. Subpoena the IT staff workers in charge of Email and the other wholly separate group that manages the IT Storage infrastructure. And ask them to produce the email or risk jail.

That's just stupid.

You can't throw somebody in jail unless you can prove that they lied. Which is not going to happen.

But, you can prove it.

That's why the "My dog ate my homework" excuse won't fly here...

Even the FBI is known for their ability to recover data that were forcibly destroyed. User their expertise.

Do that and the emails will be on their desk by morning.

They are not going to do that because they have zero reason for wanting to take away the ability to do the same. We don't have proof that a crime occured to begin with other than "We are all pretty damn sure that this story is bs." The FBI can't just go digging aorund in other agencies without any sort of court order.

You start with higher ups in the IRS looking into things, then the OIG, then some more hearings, then MAYBE the OIG calls the FBI to ask for help, then some more hearings, then some other scandal that is new and juicy comes along and/or a Republican is in the White House and the whole thing gets dropped because neither party wants this to change.

I'm going on record... if a Republican administration did this, I want them hammered just as hard.

**d-usa**

And if a Republican administration did it the Democrats would bitch and moan and put on a dog and pony show as well.

And they would make sure that they are going to be able to get away with it in the future as well.

You're a political idealist.

I'm just pragmatic and realistic.

**dogma**

whembly wrote:

5. All email servers in a professional organization use TAPE backup. Meaning if all the above fails, you can restore the server using the TAPE backups.

Ours doesn't, and that's actually considered a selling point by many of our clients. So by "all" he means "the majority of".

**Ouze**

whembly wrote:
Put the IT folks under oath.

Does everyone know that the IRS cannot lose emails, right?

Neither you nor the person in the article know the specifics of the system the IRS uses and it's all speculation. He literally does not even know what software they run, he's speculating it's Exchange.

**Grey Templar**

If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

**Ouze**

Grey Templar wrote:
If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

Although I, like Whembly and the guy from the article, am not privy to the IRS's email system, I am nonetheless quite confident in speculating it's absolutely terrible software, contracted out the lowest bidder who also was connected to someone in the appropriations process, that the specs changed constant, the program manager changed things during development without documenting anything, and it cost at least 3 times what it was supposed to while delivering 30% of the features it was sold upon.

**dogma**

whembly wrote:

We can prove it. Determine how the email system is architected. Really... it's as simple as that.

Which would result in what? Contempt of Congress? Who would be charged?

**Grey Templar**

Ouze wrote:
Grey Templar wrote:
If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

Although I, like Whembly and the guy from the article, am not privy to the IRS's email system, I am nonetheless quite confident in speculating it's absolutely terrible software, contracted out the lowest bidder who also was connected to someone in the appropriations process, that the specs changed constant, the program manager changed things during development without documenting anything, and it cost at least 3 times what it was supposed to while delivering 30% of the features it was sold upon.

It might be terrible software, but my statement still stands.

Not having tons of backups and protection against data loss would be a colossal screw up. I know the government doesn't hold a high standard, but not having those things is insane.

**d-usa**

Nothing will change, because people want to be able to keep stuff secret:

http://en.wikipedia.org/wiki/Bush_White_House_email_controversy

Both parties do it, both parties bitch about it, both parties will keep on doing it.

Outrage gets you donations and votes, that's the only reason they are even pretending to care as much as they are.

**dogma**

Grey Templar wrote:

Not having tons of backups and protection against data loss would be a colossal screw up. I know the government doesn't hold a high standard, but not having those things is insane.

All data is not created equal. For an organization like the IRS emails are way less important than financial information.

**whembly**

dogma wrote:
whembly wrote:

We can prove it. Determine how the email system is architected. Really... it's as simple as that.

Which would result in what? Contempt of Congress? Who would be charged?

Simply to determine if recovery is possible.

Automatically Appended Next Post:

dogma wrote:
Grey Templar wrote:

Not having tons of backups and protection against data loss would be a colossal screw up. I know the government doesn't hold a high standard, but not having those things is insane.

All data is not created equal. For an organization like the IRS emails are way less important than financial information.

How so? Tax payer's confidential isn't as important as financial information?

**dogma**

whembly wrote:

Simply to determine if recovery is possible.

Recovery is possible for nearly all data, the only question is how much money you're willing to expend in the course of recovery; and how much you can recover.

whembly wrote:

How so? Tax payer's confidential isn't as important as financial information?

I never said otherwise. Form data is obviously as important as other form data, why you gathered something else from my comment is a mystery to me.

**whembly**

dogma wrote:
whembly wrote:

Simply to determine if recovery is possible.

Recovery is possible for nearly all data, the only question is how much money you're willing to expend in the course of recovery; and how much you can recover.

And that's a fair point... won't argue that.

whembly wrote:

How so? Tax payer's confidential isn't as important as financial information?

I never said otherwise. Form data is obviously as important as other form data, why you gathered something else from my comment is a mystery to me.

Did I misunderstand your response to GT?

dogma wrote:
Grey Templar wrote:

Not having tons of backups and protection against data loss would be a colossal screw up. I know the government doesn't hold a high standard, but not having those things is insane.

All data is not created equal. For an organization like the IRS emails are way less important than financial information.

**Relapse**

Grey Templar wrote:
If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

Obamacare had some really excellent system designers!

**Grey Templar**

If the IRS's systems worked as poorly as the ACA we'd definitely know about it.

**whembly**

Relapse wrote:
Grey Templar wrote:
If the IRS didn't have massive numbers of backup systems and protection against this very sort of thing I'd question their competence and the mental capacity of everyone who designed their system.

Obamacare had some really excellent system designers!

Well... the point being is that we're being asked to suspend our disbelief that it only affected Lerner's emails during the timeframe in question.

If it was that catastrophic, it would've impacted more than just Lerner.

Knowwhatimean?

Automatically Appended Next Post:
EDIT:
Found out that the IRS uses Microsoft Exchange.

IRS is simply lying.

**dogma**

whembly wrote:

Did I misunderstand your response to GT?

No? I'm not sure what your objection is.

Taking a step back: the confidential information of taxpayers is not likely to be included in an email, and if it is it can be easily censored; a problem which will likely crop up in the release of IRS records regarding Cincinnati.

whembly wrote:

Well... the point being is that we're being asked to suspend our disbelief that it only affected Lerner's emails during the timeframe in question.=

No, that's wrong. The argument being made is that Lerner and "...other Administration departments..." lost their emails.