You know that scene in Airplane where the plane is skidding down the runway and everyone keeps running from terminal to terminal as it passes each one? That's how I feel on busy days here at CR when new posts keep popping up.
Of course. I'm sure Billy Joe and Bobby Sue will download it immediately and learn to manipulate it, because not even having to learn Squirrel is too much to ask when it empowers you to look at 62.5 million records.
Gretchen's suggestion makes plenty of sense to me, Tanta. Obviously she is thinking that even if the average taxpayer won't examine the records, commentators such as reporters or bloggers with a budget could do so. Probably there are valuable lessons to be learned. Sunlight is the best disinfectant.
Life may be too damn short to read every word but I read every word of Tanta's post and feel a bit younger for the effort. Laughter adds 7 years to ou life expectancy.
Nowadays the computers sold to common people are easily powerful enough to do analyzes on that kind of data. There are enough nerds that would actually be interested in doing it for free, too. The problem is, drawing conclusions from the analyses requires insight very few people have, so the nerds wouldn't know what to do with their results.
Tanta, she just wants to keep you busy reviewing 60 million mortgages so you'll stop writing nasty posts about her.
I have actually downloaded and used similar volumes of data from public-access genomic databases, with only a garden-variety desktop. It is quite a project, but doable.
Why do any analysis whatsoever? the cheap-credit end-game always puts money in the hands of those who can't repay. We'll know what all the s#!t is worth as time passes and payments are made on the loans... or not. Meanwhile, get short and stay short. There's lots of air beneath these levels and lots of players needing cash not "securities" going forward.
Gretchen also apparently asks that we peruse each loan for signals that, say, borrowers who work in nail salons probably did not net $125K, so that we could phone up Fannie and Freddie and demand buybacks. This, too, could take some time.
Steelhead: I agree. Besides IF it were a case of wanting to be GM, it would be ENVY not jealousy. Jealousy is the desire to eliminate the object of affection from one's own lover. Exactly who would Tanta seek to oust from GMs little black book?
Your posts are becoming a bit tedious these days, dear. I've been skipping most, but thought I'd pick up on this one, what with it being a rather interesting day, etc, but your ranting against the Morg is mining rather thin veins these days...
I'd really like to see you expend your analytical prowess on something far more creative, like what you might have to offer in terms of solutions to the mess these dolts have created.
And so, to change the subject, a poem that I just read in an anthology...
When Death Comes (Mary Oliver)
When death comes
like the hungry bear in autumn;
when death comes and takes all the bright coins from his purse
to buy me, and snaps the purse shut;
when death comes
like the measles-pox;
when death comes
like an iceberg between the shoulder blades,
I want to step through the door full of curiosity, wondering:
what is it going to be like, that cottage of darkness?
And therefore I look upon everything
as a brotherhood and a sisterhood,
and I look upon time as no more than an idea,
and I consider eternity as another possibility,
and I think of each life as a flower, as common
as a field daisy, and as singular,
and each name a comfortable music in the mouth
tending as all music does, toward silence,
and each body a lion of courage, and something
precious to the earth.
When it's over, I want to say: all my life
I was a bride married to amazement.
I was the bridegroom, taking the world into my arms.
When it is over, I don't want to wonder
if I have made of my life something particular, and real.
I don't want to find myself sighing and frightened,
or full of argument.
I don't want to end up simply having visited this world.
Oh Tanta are you still using Excel 2003. You poor dear, I was wondering why the mortgage pig was unrealistically so small. Can someone please send her a hacked copy of Excel 2007.
she's amazing...IF we got such a report, and we "all" read it, a stretch to be sure, and...then what? We are, happier? Smarter? better informed? what's the point? We can then be better stewards of the shitpile?
Hopefully this post won't stay on top of the blog for long because it is a disservice to the great posts of the day. This is just stupid - but the cult of Tanta will defend her to the end.
Scott writes:
Dare I say that maybe Gretchen didn't mean to provide ALL of this data for EVERY single loan, but rather to summarize it.
You now understand Tanta's manner of argument, and discourse generally: build a strawman (mis)representing the other person's postion, then show how absurd (or racist, classist, etc.) it is! The sycophantasy here eats it up every time... rah rah!
I commend Tanta for calling a well published fourth estater a dolt. The more attention brought to bear in behalf of the truth, the less potential for bloviating and obfuscating remains.
I know that Gretchen is your favorite whipping girl, but with everything going on today, I can't understand why you bothered with this post. Disappointing.
Although I do agree that it wouldn't be a good idea to post all of the records, and the ability for Billy Joe and Bobby Sue Taxpayer to use the data would be close to zero, there are many (free) ways outside of Microsoft Office to store that kind of data and analyze it for anyone who took the time to research free and open databases. As the first poster said, PostgreSQL is one, MySQL is another, and those are just the most common and popular free databases out there. Your argument that it isn't a good idea because Microsoft's tools for the average Joe aren't sufficient doesn't hold weight. I expect better from you Tanta.
You now understand Tanta's manner of argument, and discourse generally: build a strawman (mis)representing the other person's postion, then show how absurd (or racist, classist, etc.) it is! The sycophantasy here eats it up every time... rah rah!
Who could be so stupid not to realize the GSEs already publish summary data?
The latest crisis on Wall Street means more layoffs, which could cut further into luxury spending.
Niemira noted that with the top 20 percent of American households spending almost 5 times what the bottom 20 percent spend, that's not good news for the economy.
Whats that thing that eveyone says about everyone and subprime?
Why all the Tanta bashing? If something is important, it's worth repeating: GM is an underinformed, overconfident megaphone. As far as the truth and objectivity are concerned, the old grey lady is a cadaver or well done burger: pull out the meat thermometer, Tanta, I'm convinced GM's done.
No one is bashing her we just disagree and think these type of posts are useless and lead to massive cult/group think with about 30-40 die hard fans. That is all.
SR: GM is "stupid" for wanting to wade into the details of many paid-off note files. Tanta makes a new and important point: what's the point, GM?
Besides, the mistakes made in the past ten years won't be made again for another 60 to 70 years. The horses are gone and the barn doors need not be closed, let alone locked.
what you might have to offer in terms of solutions to the mess these dolts have created.
I dare say the solution will come in due time. When everybody who wants their money back asks for it and either gets it or doesn't, then that will be the "solution." Have patience.
Oh come Tanta, you know damn well the point is that F&F should open the books to the taxpayers, now their backers, to see what they have guaranteed.
By your logic the Federal government shouldn't bother to release the details of the budget because it's a document so voluminous that no one could make sense of it.
Analyzing the F&F assets would be tedious to be sure, and Gretchen is guilty of imprecision in they request for "all" loans, but the request is not a crazy one. Hopefully someone in congress will get around to investigating the portfolio of F&F to see how far they strayed.
Question: Is it possible that F&F actually adhered to the guidelines and still got burned? If they are relying on the integrity of the loan originators then how culpable are they in the whole mess? (Which is not to suggest F&F don't have some serious problems), but it may be that they were primarily guilty of not being vigilant enough in auditing the originators. Inexcusable, but there were certainly bigger abuses going on.
I really don't think data size is the main issue here. How hard would it be to put 62.5 million records on a SQL server and make the data searchable using a web app that's available to the public? Trivially easy, in fact.
puravidavid writes:
SR: GM is "stupid" for wanting to wade into the details of many paid-off note files. Tanta makes a new and important point: what's the point, GM?
Besides, the mistakes made in the past ten years won't be made again for another 60 to 70 years. The horses are gone and the barn doors need not be closed, let alone locked.
I am not sure GM meant that - she is writing to make a point to the mass public. She is writing to get them annoyed and interested in what happened so they will demand answers. She is writing so people pick up the paper and say - "yeah that is a good idea we should stupid what those jerks did."
Tanta wants to make it seem trivial and keep defending FNM. Sometimes you need to read between the lines. Look at all the stupid supportive comments Tanta has received on this posts - its a stupid post no way around it.
You mean the Pulitzer-Prize winning reporter Gretchen Morgenson? The one who happens to be a fine-looking <a href="http://www.nytimes.com/ref/nyregion/08beat_reporting.1.html> blonde? The same Gretchen Morgenson who also worked
for Money, Forbes, Worth and Vogue magazines and who, in 1995, served as press secretary for presidential candidate Steve Forbes? Is that the one we're talking about?
Normally I love your posts, but now you're not talking about your domain (mortgage origination and securitization) but mine (information technology) -- and you sound like a fool: 62.5 million just isn't that big of a number. Even table scans over 62.5 million rows aren't going to hurt much; many (many) machines could hold an entire such table in memory. You don't like the rampant amateurism among those who talk about MBSs -- so I'm sure you'll understand when I ask that you kindly leave the data manipulation and information management to the professionals. (Who, I hasten to add, would love nothing more than to chew on that 62.5 million row database.)
Actually, on consideration, I think making the loan data public is essential now that the GSEs have been de facto nationalized. There are at present enormous possibilities for mismanagement and, worse, for deliberate mismanagement for short-term politicals goals (by any President anytime, not just Bush now). If we have to wait for the GSEs to report summarized default rates to fix loose lending, we could easily be staring at another 200 billion bill for the taxpayers. The best control would be for anyone interested to poke around in the GSE files and see if their standards are inadequate. While J6pack can't manage that volume of data, there are many who can and can produce summaries even J6pack can use.
Bottom line: the the public has to pay for the GSEs now and they deserve to know what's going on NOW, not with a 2-year lagtime.
I admit, I am confused. Does GM feel the Federal Housing Finance Agency is inadequate to the task of conducting a review of Fannie and Freddie? Does she think the FHFA is hiding something?
Otherwise, why suggest opening the books to taxpayers?
Yes, detailed data is essential now. In the same way, I demand instant access to each commenters' tax returns. After all, your tax deductions are really the same as liabilities.
I don't understand - all the data IS publicly available, and the parts that are not rolled up in easily digestible portions have already been digested by third parties and provided for a fee on a monthly basis.
However, you're correct about it not being very useful to the typical consumer. RMBS, and CMBS data to a much greater degree, has never lacked from availability of data, it has always been that its too much data for most investors to digest.
Both companies provided bucketed versions of the data in their regular quarterly filings though and in every prosup for every securitization.
FT Woods - yes by your logic we need more government agencies to review the quasi-government agency that got us into a lot of these problems. that is perfect logic - welcome to the cult.
Arun,
I've been lurking here for a few years...That poem by Mary Oliver might be a little out of place. I don't think you meant to post it in poor taste.
I just think you're unaware of Tanta's other battles?
Really now SR. "Cult"? Ad homonym attacks undercut your suggestion of valueless posts by Tanta.
I for one, am not an appologist for FNM and FRE. I audited the largest mortgage bankers in the late 70s and know what painstaking efforts were made then to assure that loan recipients were capable of discharging their obligations. Only in the past decade did the daisy chain circle jerk of originators and securitizers forget the lessons of unwarranted risk all the while listening to their own music with the lyrics supplied by "knowlegable" insiders like GM. Wake up, smell the septic tank... it ain't coffee... but you're welcome to continue drinking if you please.
Actually Freddie Mac publishes detailed loan level information on every pool it guarantees. It is available on their website, just type in a pool number and analyze away.
Deloitte & Touche has (had?) an audit contract with Fannie and Freddie. I have two friends in DQI that worked in SAS with FandF datasets; processing 5 million observations (i.e., mortgages) was not an unheard of operation.
SR: "Anything?" You further undercut your point with absolutes. I disagree with Tanta when she's off base or wrong. In this case she's not. It's pointless for GM to suggest detail post mortems that she was cheerleading as recently as last year.
I think it could be very interesting. There are some interdisciplinary folks who would both understand the mortgages and have the skills to process 62M records (or could work with people who do).
62M records isn't very big nowadays, actually. I have a database with over a 20B records and it's running on my PC fine.
maybe i'm alone in thinking...this isnt already available? obviously the idiots whose names would be in the data wouldnt have a clue what to do with the info. bur surely somebody could mine some useful unbiased info from it.
i cant sympathize whatsoever with...that's ridiculous, it's too big.
if it's too big then maybe something should be done about it. if verizon cant ever give me any decent amount of customer service because they are too big, that is pathetic.
isnt this the problem with earmarks? it's just a million here, a million there, having a bill on each piece would be too much. well those millions add up to billions.
The pooled data certainly has its uses and I thought about batch-downloading it to take a look. But, for example, looking at within-zip-code variance of appraisal changes over time might have exposed the monster fraud that's been going on before the GSEs got committed to tens of billions in fraud losses. Maybe, maybe not. But even a small chance for that kind of gain justifies a databank and allowing blogger/academics/wonks to dig into it.
Tanta has a blog. It is HER blog(& CR's). If Tanta wishes to write about GM, nasal hair, or how many assholes can dance in a flea's navel, that is most entirely up to her.
Why do some people feel that it is within their purview to "mindread" whatever Tanta's motivations were/are?
Tanta: next thread, please? And this time, no pearls before swine? They crunch-up and scarf-down the precious along with the corn and distinguish little difference.
puravidavid writes:
SR: "Anything?" You further undercut your point with absolutes. I disagree with Tanta when she's off base or wrong. In this case she's not. It's pointless for GM to suggest detail post mortems that she was cheerleading as recently as last year.
puravidavid | 09.15.08 - 5:22 pm | #
You are proving my point more than I ever could. Most people understand the spirit of what I am saying. But people like you and Tanta don't like to give anyone credit. People know what GM meant just like they know what I mean when I say CULT.
And that is why I used the word jealous, instead of envious. Even so, I concede that Arun, Li, stock-regulator and other posters forthwith will make my point more articulately.
For example, I didn't stop to consider Tanta's lame news judgment in choosing now as the time to protest the conductor's selection of "Nearer My God to Thee." Glub, glub...
And yes she can post whatever she wants but if you want to be a respected blog on economics and housing I would not keep posting on nonsense - especially with the market falling apart from the specific loans FNM helped fuel.
A single copy of the proposed document, processed on a Xerox Docutech, the fastest common laser printer, would require roughly 68 hours of processing time.
This single copy, assuming an industry standard of 4000 sheets in a case and 40 cases on a pallet, would be stored on four pallets.
Assuming industry standard "click charges", the processing alone would cost $9387.20 consuming $2268 in standard paper, or $2878 in minimum 50% post-consumer-waste recycled paper.
I'll jump in and fall somewhere in the middle. I think the point somewhat is that giving a giant dump of data like this to MOST folks, common folks, non IT folks, isn't going to accomplish much. Hence, dumb idea. However, making that data pile available to SOME IT types could be useful. But even for many of them, it would be garbage, because they wouldnt know what they heck they are looking at without a good background in what FNM and FRE have been up to historically, and also how what theyve done recently is different.
That said, if you combine someone with a lot of mortgage knowledge with a person that also has a very good DB background, well then Id think you could get something useful.
Im just not sure why this isnt available already for anyone who would really know how to get at the numbers and would want to.
So, overall, I dont really get the article's point, but that is often the case with GM, because she simplifies things to the point of nonsense.
On a theoretical level, I do think that all that data should be available to anyone, since everyone is now a "homeowner".
As far as the truth and objectivity are concerned, the old grey lady is a cadaver or well done burger: pull out the meat thermometer, Tanta, I'm convinced GM's done.
puravidavid | 09.15.08 - 4:58 pm | #
Say what you want, The Times was the first MSM I saw that definitively called LEH's bankruptcy filing yesterday afternoon: see CR's post far below.
Sorry, but I think you're off base this time. Anyone running Linux and that has basic scripting skills could do some interesting analyses on 62 million data points. Not everyone is stuck in Windows.
Just because every Joe and Jane doesn't understand structural engineering doesn't mean that it's not useful to make blueprints public after a structural disaster.
GM was clearly just trying to bring this issue to the public light - but again keep posting dorky comments about how long it would take to analyze each individual mortgage. Keep obfuscating. That is what CULTS do right?
What's the cult nonsense? Can't people disagree with a particular point by Tanta without saying she's some kind of Jim Jones? What does it add to the discussion to accuse posters here of being cultists?
Well, it's a good way to derail the discussion into flame- and troll-wars.
SR: Keep obfuscating... that is what cults do, right?
Right. Keep defending GM as trying to bring this issue to light. She's often a smug over confident gotcha player who loves to say I told you so, even when she never told us.
Wow my little niche. I run the RBC for FHFA, and even working just with pool level data, the GSE's information is gigabites. Since we give away the model to start with it would be pretty easy for IT types to modify it and work with the data set. (Swinging a big enough sun system to run in a reasonable time might be more of a challenge). It takes 45+ minutes on our 32 processor system.
Can we all get back to work now? Seems there is a global financial melt down in process; credit destruction on a pace to dwarf the deflation of the 1930s commencing.
Fair Economist writes:
What's the cult nonsense? Can't people disagree with a particular point by Tanta without saying she's some kind of Jim Jones? What does it add to the discussion to accuse posters here of being cultists?
Well, it's a good way to derail the discussion into flame- and troll-wars.
Fair Economist | 09.15.08 - 5:40 pm | #
I am saying that this blog suffers from MASSIVE group think on certain posts by Tanta. She has a cult like following that will just agree with her and not question it. I am not flaming anyone I just think these type of posts hurt the credibility of this blog.
Both companies provided bucketed versions of the data in their regular quarterly filings though and in every prosup for every securitization.
Let's count how many people whining about "transparency" have analyzed the data already publicly available and found it insufficiently detailed for them?
I for one would welcome the possibility for journalist and other people to pour into the portfolio and SEARCH within it after their chosen criteria. The former CEOs will not like it, because more shady activity might be found, maybe upon insider tips, but now with PROOF.
Markel - the companies didnt file financials for like years. Come on. Let the public see how these guys bought all the crap loans. What are you so afraid of????
Fair Economist - see how can you deal with all these opinions that seem to support Tanta's post for no real reason? What possible reason can you have for not looking over the data? Yet they just agree with her to agree - its annoying. It is cult like behavior. Are they members of an actual cult? I doubt it but that isn't the point.
BitTorrent to the rescue! What's 62.5 million records, even if not compressed? Just over past two months I helped distribute two packs (10GB and 15GB) of archival videos without any trouble whatsover. It took less than a week each.
Technology is not a problem here. There may be some implications for the privacy. But the technology part will be comfortably handled by an average 15-yeard-old wannabe system administrator.
I really wich I knew what your personal problem with Morgensen was. Trust me - it shows big time.
Only an idiot would have not recognized that the whole piece was written in irony and sarcasm with the target being how the US taxpayer is getting stuck with obligations that have never been discussed in the legislative process and are tossed onto the taxpayers bill in secret meetings.
Do try to get your personal venom under control.
I had read the article and understood the irony and the point.
Mmm, I don't think my personal IRS tax return data are subject to FOIA requests by random journalists. Why would my mortgage data be, if I had one?
I would think that there would at least be a requirement to anonymize the data before turning it over to random journalists. It's been a few years since I looked at privacy law, but the gov't has greater obligation to protect personal data then anyone does in the private sector. Is the new gov't owned gse sector public enough to trigger the higher level of privacy protection? I sort of hope so.
If nothing else, such freedom of access to records is a bloody invitation to identity theft.
I am saying that this blog suffers from MASSIVE group think on certain posts by Tanta.
Insofar as that's true, accusing the groupthinkers of being a cult is far less likely to produce a change than countering the arguments does, and much more likely that any cogent counterarguments will be buried in venomous and uninformative exchanges. There are a number of posters above discussing how they could use the kind of data GM was (possibly sarcastically) suggesting be available. Do you think the cultist charges and resulting defenses make it more or less likely a casual browser will get to those arguments?
Copout. You can't have the data because you can't handle the data? Right....
Sorry, there are many ways to provide the data that can easily be handled by 'PCs' and rather simply too.
She's obviously asking that question because she doesn't trust the answers we are getting or have gotten from this institution which we the people now own. Though I'm sure she as everyone else knows that they already have detailed stats on everything. That's how they kept making such "lucrative" business decisions.
I am not here to change minds. That is a losing proposition.
Just remember when you are gathering infomation you are only as good as the opinions around you. I am highlighting that certain opinions are biased and out of touch. That is all.
The CULT references simple make it easier for people to understand what I am saying and enjoy readind the post.
part of the problem in analyzing 65,000,000 files is that you don't have permission to invade the privacy of any of the people behind those files. Sure, some numbers may look fishy but until you examine the original loan documents and verify the original claims made during the loan application process, you cannot make any useful conclusions about the propriety of any one loan. What was lacking then and is still lacking is sufficient numbers of skilled loan officers making good judgements. The rush to profits on the part of senior management overrode any fuss that came out of the back office.
Please stop with the stupid privacy arguments. Just take all personal information out. Really you can't say they already provide this info on one hand and then on the other say if they provide it there will be massive invasion of privacy. It's just stupid.
I have to agree with those saying that this is a silly blog entry; yes, lots of us could do useful things with 63 million or so mortgages, and one could fit at least a page of info on each mortgage on a single BDROM. Granted, making available high-resolution copies of 62e+6 original loan documents packages would be dumb, but they wouldn't be analyzable anyways. I'd be surprised if there was more than a few terabytes of analyzable data at all.
As a large-scale data modeler (please, hold the boos), I would LOVE to slice-and-dice the 65M raw data records. Summary data can hide a lot of sins.
As for the suggestion of "conversion" to PDF format, it reminds me of a common technique used by scientists when forced to publish data they really don't want to publish: convert complicated tables to PDF, preferably rotated and broken across multiple pages to make scanning back into a useful numeric format cruelly difficult. Sometimes summer interns are hired to re-type the tables as part of their purported scientific "education"... oy veh...
Actually surely the suggestion was figurative and not literal? Someone with fiduciary responsibility vis-a-vis taxpayers, e.g. Treasury accountants and financial analysts, should over the coming weeks vet the books/balance sheets of FNM and FRE.
Well, THIS taxpayer has terabytes of storage and a cluster of over 150 quad-processor servers at his fingertips. I'd love to crunch on that data myself.
Loan Performance which covers a good bit of subprime, Alt A and Whole Loan Prime, covers about 25 million loans, with monthly updates going upwards of a billion rows. With minimal indexing, it's about 1.5 terabytes. This is within the realm of what you can by at Circuit City but any interesting queries are going to take days.
Um, Tanta, I have a great deal of respect for your work, but I think there are better uses for your tremendous talents than mocking reasonable proposals. Flooding the dark rooms of Fannie and Freddie with some sunlight isn't such a bad idea, though I don't know what it would accomplish in the end.
Last I checked, when there is a large disclosure like the proposal, the press and the blogosphere will get copies, put them in real SQL databases, and do the analysis. It's not hard. Then the press and the blogosphere report on what they find to the rest of the world. This doesn't seem worth your time to ridicule.
Downloading, dissecting and giving painstaking and scrutinizing analysis to the numbers and doc's may not be helpful to determine the magnitude of fraud and the thousands of culpable parties at every stage of manufacturing these trillions.
What is significant is that the extent of the unraveling values of assets and liabilities proceeds at and ever faster rate with only bottom guessers suggesting an end in sight.
Tanta is right. This is a major project that I would be reluctant to tackle unless I had absolutely no choice (and I have handled some HUGE e-discovery projects in my short legal career even if you do not count my number crunching for various political campaigns). There comes a point when you can have too much data and this would easily be one of those times.
"For example, I didn't stop to consider Tanta's lame news judgment in choosing now as the time to protest the conductor's selection of "Nearer My God to Thee." Glub, glub..."
Taking Peter Schaeffer estimate of 12 DVD we arrive at about 54GB total. It isn't going to fit on a BD-ROM (as per Jwhitland). But it is easily doable with BitTorrent.
Somebody with access to that data please remove the personally-identifiable information and post in on Pirate Bay. They already have over 100 torrents that are this big or bigger.
To answer the post-mortems: the original loan doc's and ALL THE SUPPORTING DOC's must be compared and confirmed with veracity. Crunching spread sheets won't get you there.
That's why evolution demanded professional independent auditing. Thanks, SarbOx. It's tedious and costly, but failing to do so is more expensive. Nature is replete with tedium... by which most stable and solid growth occurs across every realm: phyical, biological, empirical and human endeavors.
Good luck without paying attention to the details. See how well that's turning out?
So how are Billy Joe and Bobby Sue Taxpayer going to examine a database of 62.5 million loans? - Tanta
62 million loans to check out? I bet Mortgage Pig could do it and over night to boot. Bobby & Sue can still trust Mortgage Pig even if they can't trust Wall Street...
I think you are finding fault in the details rather than the goal. Taxpayers (or their representatives) deserve to know exactly (and I mean exactly, not in summary) how deep the doo doo is since we and our children will be cleaning up this mess.
Also, on a different note, I suggest you research team-building and personality types.
I worked a company with huge conflicts among the management team. The company shrank to a remnant of its former self. Apparently, the company that was handling our payroll (Kimstaff?) offered some team-building seminar. When we got the part about identifying personality types things became clear.
For example, strong leadership/entrepreneur/action-oriented types really dislike talking to the detail-oriented/conscientious/analytical types because they want straight info in brief, right now, and usually that is at odds with someone who wants to deliver complete and accurate information with dependencies and qualifiers.
Likewise, the detail-oriented/conscientious/analytical types detest listening to the social/persuasive/sales-and-marketing types because those types prefer persuasion over information and unfortunately are very successful at it.
Anyway, I think this might explain your disatisfaction with GM. Not to put you in a box, but your posts clearly indicate a desire to set straight the information (which often appears at odds with how GM is trying to sell a point).
Of course, I have to point out I'm speaking from the memory. I am a software engineer, not a human-resources or psycholy professional. So, I should be corrected by more knowledgeable experts if I'm in error about this.
After our company experienced the team-building seminar, there was no memo to make nice and be friendly, no incentive to get along, yet I believe the people in our workplace naturally came to accept the group dynamic once it was understood there was nothing personal in these cross-personality transactions, and that you really do want (for example) one of these social/persuasive type people interacting with the public and the clients/buyers/investors.
In an ideal world, GM would be have brains AND beauty so more of us would tune in and be accurately informed. But let's face it. Nobody has it all, not GM, not Einstein, not Obama.
I am thankful there is somebody like you to educate those of us outside the industry about all the uber-geek mortgage goodness.
Also, as someone familiar with current home computer technology, I would differ with the IT geeks who say 62.5 million records can be managed no problem on a home computer.
Say the address, price, interest rate, and other mortgage data could be stored in just 40 bytes per record. For comparison, this single sentence (uncompressed) would require 147 bytes of storage (including the period and a null terminator character).
There are 1,073,741,824 bytes in a Gibibyte so our 250,000,000 bytes are also known as 2.39 GiB (aka GB or G). My machine here at work has 2 GB of physical memory.
Microsoft Windows XP manages memory in a "virtual" fashion, so for example, it is transparent to me (as a user) that the OS does not actually have the entire file in memory all at the same time (part of it is being stored off on disc), however, it is immediately obvious to me whenever I exceed this "magic" threshold of physical memory on my machine because everything slows to a crawl as the machine must read and write out tons of memory in and out to and from the hard disc.
So, it is fair to say data of this magnitude would be unwieldy on a personal computer. Okay IT guys, go ahead and have at it. Maybe I missed something or miscalculated.
What interesting questions are you going to ask the data with all the personal information stripped out?
We already know that many but not all stated incomes were false. We already know many but not all appraisals were inflated. That is why Fannie went from a profit of several thousand million in 2006 to a loss of a few thousand million in 2008.
Without the ability to verify the data in individual loans, what else can you find out?
I work all day, and get half-drunk at night.
Waking at four to soundless dark, I stare.
In time the curtain-edges will grow light.
Till then I see what's really always there:
Unresting death, a whole day nearer now,
Making all thought impossible but how
And where and when I shall myself die.
Arid interrogation: yet the dread
Of dying, and being dead,
Flashes afresh to hold and horrify.
The mind blanks at the glare. Not in remorse
- The good not done, the love not given, time
Torn off unused - nor wretchedly because
An only life can take so long to climb
Clear of its wrong beginnings, and may never;
But at the total emptiness for ever,
The sure extinction that we travel to
And shall be lost in always. Not to be here,
Not to be anywhere,
And soon; nothing more terrible, nothing more true.
This is a special way of being afraid
No trick dispels. Religion used to try,
That vast, moth-eaten musical brocade
Created to pretend we never die,
And specious stuff that says No rational being
Can fear a thing it will not feel, not seeing
That this is what we fear - no sight, no sound,
No touch or taste or smell, nothing to think with,
Nothing to love or link with,
The anasthetic from which none come round.
And so it stays just on the edge of vision,
A small, unfocused blur, a standing chill
That slows each impulse down to indecision.
Most things may never happen: this one will,
And realisation of it rages out
In furnace-fear when we are caught without
People or drink. Courage is no good:
It means not scaring others. Being brave
Lets no one off the grave.
Death is no different whined at than withstood.
Slowly light strengthens, and the room takes shape.
It stands plain as a wardrobe, what we know,
Have always known, know that we can't escape,
Yet can't accept. One side will have to go.
Meanwhile telephones crouch, getting ready to ring
In locked-up offices, and all the uncaring
Intricate rented world begins to rouse.
The sky is white as clay, with no sun.
Work has to be done.
Postmen like doctors go from house to house.
On this one you seem to have strayed from your area of expertise, but that's good because it's a great opportunity to highlight just how technically feasible Morgenson's proposal is.
Computing power, storage, bandwidth are not remotely the problems here.
What is likely to keep this from happening is Congressional nonfeasance (props to Barry Ritholtz for that term) and REIT lobbying, including Fannie/Freddie's.
(on that last one, if anyone believes that the recent law about them not being allowed to lobby is going to be respected, please contact me for some AWESOME bridge opportunities!)
If, by some miracle, this data did become available, the crucial ingredient to making it useful would not be CPU cycles and database tables. It would be - cough, cough - the willingness of people with domain knowledge and an honest agenda to point the database nerds in the right direction.
@Lurking Larry: either you are using bad software or you are just bad programmer.
With sensible software the AP (analytical processing) queries essentially run at the speed of streamed reading from the hard drive.
We aren't talking here about OLAP or SQL. The data set is static, so there's no use of any advanced database technology or indexing. Just plow through the file record by record and as many times as needed.
However clearly you are pointing to the fact that many people may not be aware of the proper methods to process static data sets or simply not familiar with required software tools. And here I fully agree with you. But the problem isn't technical, it's the problem of personnel and/or education.
I could care less about arguing the practical details of this particular post. I follow Tanta for her wit and style and curmudgeonly disposition. Good writers don't have to be correct to be enjoyable. That's why no matter how correct certain of you may be, (and I'm not conceding any of you are) you'll never be able to hold a candle to Tanta's prose. Your writing is offensive and boorish - regardless of its technical merit.
One problem with Gretchen is that she's too saturated in sweetness and lightness to handle serious subjects. Her style is weak. When she tries to be tough it comes across as silly because its unnatural. Most of the NY Times suffers from this.
Boo on you, Tanta. And the "tranches" actually ARE the loans, too--or part of them. The "tranches" can't be rated without knowing if the houses are there, or are habitable, or whatever. So there!
why not have the database online and let one query with a web front-end the same way access to the patent/trademark database is handled? those that want access to the raw data can purchase a dvd rom for cost and import into whatever database they wish. mssql, postgress, oracle and countless other dbs can easily query 100M+ records
@Lurking Larry: And my point was that you don't want to cripple yourself by using Excel or Access.
US government routinely sells the data of similar nature and quantity to the private industry. They use very basic fixed field fixed record-length text files. Typically there are about 928 files in such a data set, one file for each SCF (first 3 digits of zip code). Due to extremely repetitive nature of those files they compress very well using standard compression utilities like PKZIP or WinZIP.
The tools to analize such files are available both for free and for money. But you have to google for them a little deeper than you would for Excel or Access.
Gretchen's suggestion makes plenty of sense to me, Tanta. Obviously she is thinking that even if the average taxpayer won't examine the records, commentators such as reporters or bloggers with a budget could do so. Probably there are valuable lessons to be learned. Sunlight is the best disinfectant.
Fannie and Freddie both offer detailed financials - they show first profits and losses, and while I did not review in past, I don't recall massive losses
Second - the current financials show portfolio breakdown and quality along woth performance by segment - and in Q1 and Q2 2008 - when you take out the fair market writedowns - there were not huge losses
In fact - from memory - if I recall my review Fannie, if you use only the liquid/cash reserves (not the tax credits on future earnings) and added 25% of so to the existing losses - they could go years before running out of money
If alleged experts like Gretchen want to review performance they could and SHOULD start with reviewing the detailed existing available financial information and data - and presenting an intelligent meaningful real world review of the financial strength or weaknesses
It is my understanding neither Fannie or Freddie ever had significant investment, exposure or losses in subprime. The currently have comparatively no exposure to subprime risk.
They also have a limited amount of ALT-A exposure - just over 10% of Fannie's current book if I recall. And they have all but stopped adding ALT-A as well
So Fannies current picture - and portfolio - seems it represents the worst case exposure for them
I think Tanta is correct ... these alleged experts in the media refuse to do any intelligent detailed review on existing available data and information - why would they do anything different with even more data and info?
And quite honestly what does it matter? The public ally reported data splits out by loan type - what benefit is there to go to a loan level revile?
Especially when if I recall correctly Fannie has never had huge losses in subprime etc in pasty or to date ...
Again Tanta is right here - the MSM refuses to make even a nominal effort it seems to do even the most rudimentary review of the financial - of facts - for any of these situations
Had they, in my opinion, we would all be better informed and if, as I believe, the data shows things to be far better than the perception, we might not be in such a death spiral right now ...
The point is NOT that the average person would know what to do with the data. The point is that experts could analyze the data openly, claims could be verified, and inconsistencies brought to light.
It's the same way that science works. You have to publish your results and data openly so it can undergo peer review. I'd like F&F to be peer reviewed.
And by the way, all the requested data could fit into 8 bytes per loan, for a total file size of about 500 MB for all the loan data. That's a very reasonable file size, amenable to analysis.
Unfortunately the one who "dosen't get it here" isn't Gretchen. Modern databases, like the ones used at Freddie and Fannie, can plow through 62.5 million "rows" (if that's the limit of your understanding of data) in no time.
I concur what @reason wrote: "for a total file size of about 500 MB" This is very true. The data we talk about is extremely repetitive. If instead of using simple WinZIP we used one of many specialized compression tools (e.g. based on PATRICIA-trees) the data would fit on the CD-ROM, not even a DVD would be reqiuired.
The commercial tools to analyze such data sets were available since the CD-burners became popular and ran efficiently on the typical office computers from the same era. All that predates even the use of DVD and DVD-burners.
Again, technology is not a problem. The knowledge is (or rather lack of it.)
renter writes:
tanta is getting tiresome. I simply skip most of her posts. As an IT professional, I would love to have that data, to form my own conclusions.
What is getting tiresome are people who would rather kick the scapegoat rather than analyze the numbers.
Get the data yourself.
Freddie Mac already does a loan level disclosure. Freddie Mac News Center a..._loanlevel.html
I'm coming in very late on this, but I think what GM is arguing is that one cannot trust the "summary" statements we've been seeing for the past few years to be truthful. Assuming that's true (and, to state my own bias, I think that's clearly been the case), what's the solution other than "full disclosure"? Granted, I would not slog through a hundred million datasets, but somebody would, preferably somebody without a direct financial interest in making it appear that an entity's numbers look awesome. Such analysis might have (I would actually argue would have, but I try to dial back my own enthusiasm) prevented the ongoing plunge from the stratosphere we're currently witnessing...
ever pays to look just at the paper; that's how things got so bad. Look at the actual thing. Go out and look at the houses. "The map is not the territory."--Alford Korzybski
While I normally hang on Tanta's every word, I'm with G.M. on this one too. Now that Frannie are firmly under the government's wing especially, more transparency is better. There is a market for the information and technically it's not intractable. Get it out there in a format people can really use, and folks will start to use it, and those who do will do value-added analysis that will be worth a lot of money to taxpayers and investors.
More importantly, now that we've given this administration and its successor the ability to fiddle with Frannie, there's a high likelihood we'll regret not having that transparency, because it'll breed scandals...
the willingness of people with domain knowledge and an honest agenda to point the database nerds in the right direction.
What of people who are both database nerds and replete with domain knowledge? Sasquatchian? We're slaves to 'da man'?
Or we're robots who don't pay any taxes? We're just part of a vast conspiracy? Riiiiiiiiiiiiiiiight.
Yep, I agree with GM and sotck regulator, we need that data. I'm sure many wonderful things can be done with it. I myself can't figure out any use to me personally, but with that much data surely we can prove or disprove the existence of God. It's the mortgage equivalence of the hadron collider. Lots and lots of data for the conspiracy minded and something for the IT guys to do on a Saturday night.
I can't seem to do anything with the information I've got, but with more data points I will be able to prove, beyond a shadow of a doubt, that large corporations aren't always run at optimum efficiency and that politicians are swayed by money. WE NEED TO PROVE THIS, STAT!
Though not explicitly stated, I assume she's speaking about the retained portfolio. Your claims about the inability to account for those loans because of size are incorrect. Freddie has been posting the details of all MBS loans issued since late 2005, a lot of data. There are systems that deal in terabytes that easily could handle such a project. I know because I work with them. Have Gretchen give me a call.
I downloaded the AIS .pdf file in your link. I think your point is the government routinely delivers large databases to its clients.
Sorry I didn't point this out before, but it appears you and others have focused on the statement Tanta made about delivery of information on magnetic tape (which, by the way, I believe was a snarky comment and not a guess about a future government distribution plan). I never said the data could not be compressed, or that it couldn't be delivered on CD, DVD, or torrent. I agree it could.
My posts have focused on affirming the limitations Tanta suggested about using Excel or Access to examine such data. For example, my estimate of the uncompressed database is 2.39 GB and Tanta mentioned some versions of Access can not support single files beyond 2 GB.
So, in defense of Tanta, it seems unlikely Billy Joe and Bobby Sue Taxpayer are going to examine a database of 62.5 million loans using Excel and Access. But this is moot anyway, because I do not believe Billy Joe and Bobby Sue need to examine the database themselves so long as someone without an agenda does (and gets the word out.)
I would love to have loan-level time-of-origination data for the GSEs, but even that's not enough. I'd kill for monthly delinquency status on those loans, like that provided in LoanPerformance.
Its clear she didn't think this through. But I am reminded here of an English Professor who is criticizing the Grammar of the proof of Fermat's last theorem. She is requesting more transparency. How about a simple well done random sample of a class of loans made. Complete detail on the sample pulled double blind from the database. Whats the big dieal?
Might I recommend PostgreSQL? It can easily accommodate 62.5 million records, and is fast and free.
Access is for suckers.
You know that scene in Airplane where the plane is skidding down the runway and everyone keeps running from terminal to terminal as it passes each one? That's how I feel on busy days here at CR when new posts keep popping up.
Tanta
You're swelll, but do you realize that you come across as jealous?
Think of it as a WPA project for the 21st century.
Maybe, as real punishment, they should make them hand-write that report. On hemp paper, using those old-school inky fancy metal-tipped pens.
Might I recommend PostgreSQL?
Of course. I'm sure Billy Joe and Bobby Sue will download it immediately and learn to manipulate it, because not even having to learn Squirrel is too much to ask when it empowers you to look at 62.5 million records.
Gretchen's suggestion makes plenty of sense to me, Tanta. Obviously she is thinking that even if the average taxpayer won't examine the records, commentators such as reporters or bloggers with a budget could do so. Probably there are valuable lessons to be learned. Sunlight is the best disinfectant.
Geez Tanata,give GM a break,at least she doesn't still beat her wife.
Right on Stalky, but I have a better idea - Omit the names and personal stuff and send all those personal records to her offices in hard copy format.
But Tanta,think of the number of pigs you could draw on 950 spreadsheets!
Microsoft was thinking ahead and now has 1048576 rows per sheet in 2007. This brings it down to a more manageable 60 sheets.
How many mortgage pigs can you fit in 1048576 rows?
Life may be too damn short to read every word but I read every word of Tanta's post and feel a bit younger for the effort. Laughter adds 7 years to ou life expectancy.
I just know I wouldn't want try to manipulate one on my home PC; I'd guess that it wouldn't exactly run very fast.
Thats cause you didn't take my advice last time you're computer had a cold and get a Mac
Its funny after about half of the first paragraph all I could think was that must be tens of millions of loans...
And the only stuff I know about mortgages etc is from coming here to CR and the occasional Wiki visit...
........
Tanta:
You rock!!!
Kind of "being in the business" comment:
Nowadays the computers sold to common people are easily powerful enough to do analyzes on that kind of data. There are enough nerds that would actually be interested in doing it for free, too. The problem is, drawing conclusions from the analyses requires insight very few people have, so the nerds wouldn't know what to do with their results.
At the end (spoiler alert!) it turns out that Gretchen is ironically suggesting that we eat Irish babies!
Wait, no...
sorry, that was "A Modest Proposal." My bad.
And, ask her for her findings...if she refuses the news headline is "Journalist refuses to research story details !"
Tanta always engages in cat fight w/ gretchen!!!
My other modest proposal is that we eat borrowers who are underwater. Oh, and bloggers who nit-pick about things like, say, accuracy.
Magnificant mouse clic money monday!
The books have been misplaced. That's the problem...
Tom Stone writes:
Geez Tanata,give GM a break,at least she doesn't still beat her wife.
Does Tanta?
This is news to me.....
.....
Not so bad Tanta:
Wouldn't you like to have the opportunity to analyze your neighbour's mortgage situation.
Well she's a JOURNALIST, is she not? And come to think of it, that was what Palin studied to be. Must be catching.
I watched her try desperately to form a coherent thought on Charlie Rose.
It was like watching a chimpanzee discover a mirror for the first time.
Tanta, she just wants to keep you busy reviewing 60 million mortgages so you'll stop writing nasty posts about her.
I have actually downloaded and used similar volumes of data from public-access genomic databases, with only a garden-variety desktop. It is quite a project, but doable.
Thanks for a laugh after a rather startling 24 hours . . .
I dunno. There is so much going on right now, Gretchen does not interest me so much.
Why do any analysis whatsoever? the cheap-credit end-game always puts money in the hands of those who can't repay. We'll know what all the s#!t is worth as time passes and payments are made on the loans... or not. Meanwhile, get short and stay short. There's lots of air beneath these levels and lots of players needing cash not "securities" going forward.
Dare I say that maybe Gretchen didn't mean to provide ALL of this data for EVERY single loan, but rather to summarize it. E.g., you'd say:
There were X 30 year fixed loans insured in 1998 between 150,000 and 175,000 of value with ...
Ok, wait, I went back and re-read what she said. Yes, this is pretty ludicrous.
GettingevenwithCNBC:
Your comment about Tanta being jealous shows an extreme lack of intelligence. You need to apologize.
Come on Tanta, you're gonna give Gretchen Morgenson a complex. No wait. . . . . too late, she already has one.
Gretchen also apparently asks that we peruse each loan for signals that, say, borrowers who work in nail salons probably did not net $125K, so that we could phone up Fannie and Freddie and demand buybacks. This, too, could take some time.
We could then interview each borrower!
Tanta: This idea of GM has its values.
Have a look at all the projects Carl Malamud did to publish this kind of public information, see
Public.Resource.Org
Sounds like there would be plenty
of public interest behind this!
Think of all the politicians we would find out about 8-}
Steelhead: I agree. Besides IF it were a case of wanting to be GM, it would be ENVY not jealousy. Jealousy is the desire to eliminate the object of affection from one's own lover. Exactly who would Tanta seek to oust from GMs little black book?
LAMP
Free as in speech!
While GM's idea is indeed a stupid one, there's no issue about taxpayer's having the proper software.
You put the database on a server, and then have taxpayers access the DB through e.g. a web interface.
Tanta
Your posts are becoming a bit tedious these days, dear. I've been skipping most, but thought I'd pick up on this one, what with it being a rather interesting day, etc, but your ranting against the Morg is mining rather thin veins these days...
I'd really like to see you expend your analytical prowess on something far more creative, like what you might have to offer in terms of solutions to the mess these dolts have created.
And so, to change the subject, a poem that I just read in an anthology...
When Death Comes (Mary Oliver)
When death comes
like the hungry bear in autumn;
when death comes and takes all the bright coins from his purse
to buy me, and snaps the purse shut;
when death comes
like the measles-pox;
when death comes
like an iceberg between the shoulder blades,
I want to step through the door full of curiosity, wondering:
what is it going to be like, that cottage of darkness?
And therefore I look upon everything
as a brotherhood and a sisterhood,
and I look upon time as no more than an idea,
and I consider eternity as another possibility,
and I think of each life as a flower, as common
as a field daisy, and as singular,
and each name a comfortable music in the mouth
tending as all music does, toward silence,
and each body a lion of courage, and something
precious to the earth.
When it's over, I want to say: all my life
I was a bride married to amazement.
I was the bridegroom, taking the world into my arms.
When it is over, I don't want to wonder
if I have made of my life something particular, and real.
I don't want to find myself sighing and frightened,
or full of argument.
I don't want to end up simply having visited this world.
Stalky wrote, Might I recommend PostgreSQL?
Damn straight! Go PG!
Oh Tanta are you still using Excel 2003. You poor dear, I was wondering why the mortgage pig was unrealistically so small. Can someone please send her a hacked copy of Excel 2007.
she's amazing...IF we got such a report, and we "all" read it, a stretch to be sure, and...then what? We are, happier? Smarter? better informed? what's the point? We can then be better stewards of the shitpile?
Hopefully this post won't stay on top of the blog for long because it is a disservice to the great posts of the day. This is just stupid - but the cult of Tanta will defend her to the end.
SR
Tanta,
I know you and Gretchen are actually best friends. This is just the CPA's version of Penn and Teller, right?
Scott writes:
Dare I say that maybe Gretchen didn't mean to provide ALL of this data for EVERY single loan, but rather to summarize it.
You now understand Tanta's manner of argument, and discourse generally: build a strawman (mis)representing the other person's postion, then show how absurd (or racist, classist, etc.) it is! The sycophantasy here eats it up every time... rah rah!
I commend Tanta for calling a well published fourth estater a dolt. The more attention brought to bear in behalf of the truth, the less potential for bloviating and obfuscating remains.
Go, Auntie Tanta, go!
Cult Membership accepted
I know that Gretchen is your favorite whipping girl, but with everything going on today, I can't understand why you bothered with this post. Disappointing.
Although I do agree that it wouldn't be a good idea to post all of the records, and the ability for Billy Joe and Bobby Sue Taxpayer to use the data would be close to zero, there are many (free) ways outside of Microsoft Office to store that kind of data and analyze it for anyone who took the time to research free and open databases. As the first poster said, PostgreSQL is one, MySQL is another, and those are just the most common and popular free databases out there. Your argument that it isn't a good idea because Microsoft's tools for the average Joe aren't sufficient doesn't hold weight. I expect better from you Tanta.
You now understand Tanta's manner of argument, and discourse generally: build a strawman (mis)representing the other person's postion, then show how absurd (or racist, classist, etc.) it is! The sycophantasy here eats it up every time... rah rah!
Who could be so stupid not to realize the GSEs already publish summary data?
OT but it looks like this post could use it:
The latest crisis on Wall Street means more layoffs, which could cut further into luxury spending.
Niemira noted that with the top 20 percent of American households spending almost 5 times what the bottom 20 percent spend, that's not good news for the economy.
Whats that thing that eveyone says about everyone and subprime?
.....
So how are Billy Joe and Bobby Sue Taxpayer going to examine a database of 62.5 million loans?
Prison labor; 2.3 million prisoners, 4 reviews per day, 50 cents per day = 780K
some of the newer convicts will be very familiar with loan docs.
Its a win win dear
It's a STUPID post.
What is so bad about seeing how off track FNM and FRE got? I don't care if it is every stupid lone - publish the data.
Tanta is too worried about defending these 'great' companies.
If a new reader comes to this blog and sees this as the first post, they will think this is a silly blog and it is not.
But again the CULT will tell her how great she is.
SR
CNNMoney.com: 404 Page Not Found
Why all the Tanta bashing? If something is important, it's worth repeating: GM is an underinformed, overconfident megaphone. As far as the truth and objectivity are concerned, the old grey lady is a cadaver or well done burger: pull out the meat thermometer, Tanta, I'm convinced GM's done.
I just know I wouldn't want try to manipulate one on my home PC
i wouldn't want try to that either.
No one is bashing her we just disagree and think these type of posts are useless and lead to massive cult/group think with about 30-40 die hard fans. That is all.
SR: GM is "stupid" for wanting to wade into the details of many paid-off note files. Tanta makes a new and important point: what's the point, GM?
Besides, the mistakes made in the past ten years won't be made again for another 60 to 70 years. The horses are gone and the barn doors need not be closed, let alone locked.
what you might have to offer in terms of solutions to the mess these dolts have created.
I dare say the solution will come in due time. When everybody who wants their money back asks for it and either gets it or doesn't, then that will be the "solution." Have patience.
Oh come Tanta, you know damn well the point is that F&F should open the books to the taxpayers, now their backers, to see what they have guaranteed.
By your logic the Federal government shouldn't bother to release the details of the budget because it's a document so voluminous that no one could make sense of it.
Analyzing the F&F assets would be tedious to be sure, and Gretchen is guilty of imprecision in they request for "all" loans, but the request is not a crazy one. Hopefully someone in congress will get around to investigating the portfolio of F&F to see how far they strayed.
Question: Is it possible that F&F actually adhered to the guidelines and still got burned? If they are relying on the integrity of the loan originators then how culpable are they in the whole mess? (Which is not to suggest F&F don't have some serious problems), but it may be that they were primarily guilty of not being vigilant enough in auditing the originators. Inexcusable, but there were certainly bigger abuses going on.
I really don't think data size is the main issue here. How hard would it be to put 62.5 million records on a SQL server and make the data searchable using a web app that's available to the public? Trivially easy, in fact.
puravidavid writes:
SR: GM is "stupid" for wanting to wade into the details of many paid-off note files. Tanta makes a new and important point: what's the point, GM?
Besides, the mistakes made in the past ten years won't be made again for another 60 to 70 years. The horses are gone and the barn doors need not be closed, let alone locked.
I am not sure GM meant that - she is writing to make a point to the mass public. She is writing to get them annoyed and interested in what happened so they will demand answers. She is writing so people pick up the paper and say - "yeah that is a good idea we should stupid what those jerks did."
Tanta wants to make it seem trivial and keep defending FNM. Sometimes you need to read between the lines. Look at all the stupid supportive comments Tanta has received on this posts - its a stupid post no way around it.
SR
You mean the Pulitzer-Prize winning reporter Gretchen Morgenson? The one who happens to be a fine-looking <a href="http://www.nytimes.com/ref/nyregion/08beat_reporting.1.html> blonde? The same Gretchen Morgenson who also worked
for Money, Forbes, Worth and Vogue magazines and who, in 1995, served as press secretary for presidential candidate Steve Forbes? Is that the one we're talking about?
CR, how much will it cost in the labor hours to satisfy her demands?
atta good way to put the tax payers money into use.
Tanta,
Normally I love your posts, but now you're not talking about your domain (mortgage origination and securitization) but mine (information technology) -- and you sound like a fool: 62.5 million just isn't that big of a number. Even table scans over 62.5 million rows aren't going to hurt much; many (many) machines could hold an entire such table in memory. You don't like the rampant amateurism among those who talk about MBSs -- so I'm sure you'll understand when I ask that you kindly leave the data manipulation and information management to the professionals. (Who, I hasten to add, would love nothing more than to chew on that 62.5 million row database.)
Actually, on consideration, I think making the loan data public is essential now that the GSEs have been de facto nationalized. There are at present enormous possibilities for mismanagement and, worse, for deliberate mismanagement for short-term politicals goals (by any President anytime, not just Bush now). If we have to wait for the GSEs to report summarized default rates to fix loose lending, we could easily be staring at another 200 billion bill for the taxpayers. The best control would be for anyone interested to poke around in the GSE files and see if their standards are inadequate. While J6pack can't manage that volume of data, there are many who can and can produce summaries even J6pack can use.
Bottom line: the the public has to pay for the GSEs now and they deserve to know what's going on NOW, not with a 2-year lagtime.
The are not demands - the piece was obviously meant to get people to THINK. Something the cult of Tanta can't do unless she tells them what to think.
Tanta what exactly is your bone with GM? She obviously reported on something you were part of?
SR
I wouldn't want try to manipulate one on my home PC
If I weren't being paid to do so, I wouldn't try to manipulate any of it on a mainframe, either.
Gretchen must hang out with a totally different circle of taxpayers than I do.
I admit, I am confused. Does GM feel the Federal Housing Finance Agency is inadequate to the task of conducting a review of Fannie and Freddie? Does she think the FHFA is hiding something?
Otherwise, why suggest opening the books to taxpayers?
wow... the IT dorks sure are a testy bunch aren't they?
Funny post!
I like reading Tanta's GM posts while thinking of Andy Rooney doing the voice. Seems like a nice fit.
Yes, detailed data is essential now. In the same way, I demand instant access to each commenters' tax returns. After all, your tax deductions are really the same as liabilities.
I don't understand - all the data IS publicly available, and the parts that are not rolled up in easily digestible portions have already been digested by third parties and provided for a fee on a monthly basis.
However, you're correct about it not being very useful to the typical consumer. RMBS, and CMBS data to a much greater degree, has never lacked from availability of data, it has always been that its too much data for most investors to digest.
Both companies provided bucketed versions of the data in their regular quarterly filings though and in every prosup for every securitization.
-Frank
FT Woods - yes by your logic we need more government agencies to review the quasi-government agency that got us into a lot of these problems. that is perfect logic - welcome to the cult.
SR
Arun,
I've been lurking here for a few years...That poem by Mary Oliver might be a little out of place. I don't think you meant to post it in poor taste.
I just think you're unaware of Tanta's other battles?
Tanta is just pissed that her expertise in mortgage banking is quickly becoming worthless.
At least she will have the qualifications to be a secretary.
Really now SR. "Cult"? Ad homonym attacks undercut your suggestion of valueless posts by Tanta.
I for one, am not an appologist for FNM and FRE. I audited the largest mortgage bankers in the late 70s and know what painstaking efforts were made then to assure that loan recipients were capable of discharging their obligations. Only in the past decade did the daisy chain circle jerk of originators and securitizers forget the lessons of unwarranted risk all the while listening to their own music with the lyrics supplied by "knowlegable" insiders like GM. Wake up, smell the septic tank... it ain't coffee... but you're welcome to continue drinking if you please.
Actually Freddie Mac publishes detailed loan level information on every pool it guarantees. It is available on their website, just type in a pool number and analyze away.
SR - why are you putting words in my mouth that I didn't write?
Deloitte & Touche has (had?) an audit contract with Fannie and Freddie. I have two friends in DQI that worked in SAS with FandF datasets; processing 5 million observations (i.e., mortgages) was not an unheard of operation.
Yes by CULT I mean there is a group of common posters on this blog that will agree with ANYTHING Tanta writes.
SR
processing 5 million observations (i.e., mortgages) was not an unheard of operation.
For Joe and Suzy Q. Taxpayer, it's unheard of.
proc whatthefuckamidoing;
set dunno; data=fannie freddie;
run;
SR: "Anything?" You further undercut your point with absolutes. I disagree with Tanta when she's off base or wrong. In this case she's not. It's pointless for GM to suggest detail post mortems that she was cheerleading as recently as last year.
I think it could be very interesting. There are some interdisciplinary folks who would both understand the mortgages and have the skills to process 62M records (or could work with people who do).
62M records isn't very big nowadays, actually. I have a database with over a 20B records and it's running on my PC fine.
maybe i'm alone in thinking...this isnt already available? obviously the idiots whose names would be in the data wouldnt have a clue what to do with the info. bur surely somebody could mine some useful unbiased info from it.
i cant sympathize whatsoever with...that's ridiculous, it's too big.
if it's too big then maybe something should be done about it. if verizon cant ever give me any decent amount of customer service because they are too big, that is pathetic.
isnt this the problem with earmarks? it's just a million here, a million there, having a bill on each piece would be too much. well those millions add up to billions.
The pooled data certainly has its uses and I thought about batch-downloading it to take a look. But, for example, looking at within-zip-code variance of appraisal changes over time might have exposed the monster fraud that's been going on before the GSEs got committed to tens of billions in fraud losses. Maybe, maybe not. But even a small chance for that kind of gain justifies a databank and allowing blogger/academics/wonks to dig into it.
That would only require a 625,812.5 page "disclosure.
That's just eveel.
To state the obvious:
Tanta has a blog. It is HER blog(& CR's). If Tanta wishes to write about GM, nasal hair, or how many assholes can dance in a flea's navel, that is most entirely up to her.
Why do some people feel that it is within their purview to "mindread" whatever Tanta's motivations were/are?
That poem was execrable.
Tanta: next thread, please? And this time, no pearls before swine? They crunch-up and scarf-down the precious along with the corn and distinguish little difference.
puravidavid writes:
SR: "Anything?" You further undercut your point with absolutes. I disagree with Tanta when she's off base or wrong. In this case she's not. It's pointless for GM to suggest detail post mortems that she was cheerleading as recently as last year.
puravidavid | 09.15.08 - 5:22 pm | #
You are proving my point more than I ever could. Most people understand the spirit of what I am saying. But people like you and Tanta don't like to give anyone credit. People know what GM meant just like they know what I mean when I say CULT.
SR
David Sternfeld:
And that is why I used the word jealous, instead of envious. Even so, I concede that Arun, Li, stock-regulator and other posters forthwith will make my point more articulately.
For example, I didn't stop to consider Tanta's lame news judgment in choosing now as the time to protest the conductor's selection of "Nearer My God to Thee." Glub, glub...
And yes she can post whatever she wants but if you want to be a respected blog on economics and housing I would not keep posting on nonsense - especially with the market falling apart from the specific loans FNM helped fuel.
But what do I know. I am not part of the CULT.
SR
Tanta, please delete this shit. I don't have the time to be a social services volunteer.
A single copy of the proposed document, processed on a Xerox Docutech, the fastest common laser printer, would require roughly 68 hours of processing time.
This single copy, assuming an industry standard of 4000 sheets in a case and 40 cases on a pallet, would be stored on four pallets.
Assuming industry standard "click charges", the processing alone would cost $9387.20 consuming $2268 in standard paper, or $2878 in minimum 50% post-consumer-waste recycled paper.
All true, but perhaps one of your readers - ah, taxpayer - may have a quantitative analytics firm and would not shy away from 100MM+ record database.
Also, perhaps that person even has banking experience with a large regional bank (although early in my career).
Hmm, I wonder who that could be. And, guess what - it WILL NOT be pro bono. Done that on an election campaign once.
Thank you, Markel
I'll jump in and fall somewhere in the middle. I think the point somewhat is that giving a giant dump of data like this to MOST folks, common folks, non IT folks, isn't going to accomplish much. Hence, dumb idea. However, making that data pile available to SOME IT types could be useful. But even for many of them, it would be garbage, because they wouldnt know what they heck they are looking at without a good background in what FNM and FRE have been up to historically, and also how what theyve done recently is different.
That said, if you combine someone with a lot of mortgage knowledge with a person that also has a very good DB background, well then Id think you could get something useful.
Im just not sure why this isnt available already for anyone who would really know how to get at the numbers and would want to.
So, overall, I dont really get the article's point, but that is often the case with GM, because she simplifies things to the point of nonsense.
On a theoretical level, I do think that all that data should be available to anyone, since everyone is now a "homeowner".
Markel writes:
Tanta, please delete this shit. I don't have the time to be a social services volunteer.
Markel | 09.15.08 - 5:29 pm | #
No you are much too busy mixing the Koolaid..
Can't wait for her next post defending WM!
SR
As far as the truth and objectivity are concerned, the old grey lady is a cadaver or well done burger: pull out the meat thermometer, Tanta, I'm convinced GM's done.
puravidavid | 09.15.08 - 4:58 pm | #
Say what you want, The Times was the first MSM I saw that definitively called LEH's bankruptcy filing yesterday afternoon: see CR's post far below.
Sorry, but I think you're off base this time. Anyone running Linux and that has basic scripting skills could do some interesting analyses on 62 million data points. Not everyone is stuck in Windows.
Just because every Joe and Jane doesn't understand structural engineering doesn't mean that it's not useful to make blueprints public after a structural disaster.
.
GM was clearly just trying to bring this issue to the public light - but again keep posting dorky comments about how long it would take to analyze each individual mortgage. Keep obfuscating. That is what CULTS do right?
Nice beat down Tanta. Let your purse swing free.
What's the cult nonsense? Can't people disagree with a particular point by Tanta without saying she's some kind of Jim Jones? What does it add to the discussion to accuse posters here of being cultists?
Well, it's a good way to derail the discussion into flame- and troll-wars.
SR: Keep obfuscating... that is what cults do, right?
Right. Keep defending GM as trying to bring this issue to light. She's often a smug over confident gotcha player who loves to say I told you so, even when she never told us.
Seems you're the cultist and septic drinker here.
Enjoy!
What a surprising response. Lesson learned, don't question the publics ability to process large amounts of data.
my one: attch it to boinc!!!
))
i run seti on my one great stuff and it uses my vast exess pc power
Wow my little niche. I run the RBC for FHFA, and even working just with pool level data, the GSE's information is gigabites. Since we give away the model to start with it would be pretty easy for IT types to modify it and work with the data set. (Swinging a big enough sun system to run in a reasonable time might be more of a challenge). It takes 45+ minutes on our 32 processor system.
Can we all get back to work now? Seems there is a global financial melt down in process; credit destruction on a pace to dwarf the deflation of the 1930s commencing.
You think?
Fair Economist writes:
What's the cult nonsense? Can't people disagree with a particular point by Tanta without saying she's some kind of Jim Jones? What does it add to the discussion to accuse posters here of being cultists?
Well, it's a good way to derail the discussion into flame- and troll-wars.
Fair Economist | 09.15.08 - 5:40 pm | #
I am saying that this blog suffers from MASSIVE group think on certain posts by Tanta. She has a cult like following that will just agree with her and not question it. I am not flaming anyone I just think these type of posts hurt the credibility of this blog.
SR
Yes and that melt down was caused by CFC, WM and FNM to name a few. All firms Tanta seems to defend. Its annoying.
SR
Both companies provided bucketed versions of the data in their regular quarterly filings though and in every prosup for every securitization.
Let's count how many people whining about "transparency" have analyzed the data already publicly available and found it insufficiently detailed for them?
Crickets.
I for one would welcome the possibility for journalist and other people to pour into the portfolio and SEARCH within it after their chosen criteria. The former CEOs will not like it, because more shady activity might be found, maybe upon insider tips, but now with PROOF.
Markel - the companies didnt file financials for like years. Come on. Let the public see how these guys bought all the crap loans. What are you so afraid of????
SR
Fair Economist - see how can you deal with all these opinions that seem to support Tanta's post for no real reason? What possible reason can you have for not looking over the data? Yet they just agree with her to agree - its annoying. It is cult like behavior. Are they members of an actual cult? I doubt it but that isn't the point.
SR
BitTorrent to the rescue! What's 62.5 million records, even if not compressed? Just over past two months I helped distribute two packs (10GB and 15GB) of archival videos without any trouble whatsover. It took less than a week each.
Technology is not a problem here. There may be some implications for the privacy. But the technology part will be comfortably handled by an average 15-yeard-old wannabe system administrator.
Tanta
That was SARCASM.
That was IRONY.
I really wich I knew what your personal problem with Morgensen was. Trust me - it shows big time.
Only an idiot would have not recognized that the whole piece was written in irony and sarcasm with the target being how the US taxpayer is getting stuck with obligations that have never been discussed in the legislative process and are tossed onto the taxpayers bill in secret meetings.
Do try to get your personal venom under control.
I had read the article and understood the irony and the point.
Ann - they don't care. They agree with most everything Tanta writes.
It is clear as day GM is trying to YELL hey watch out tax payer they are trying to screw you.
But last week Tanta told us FNM had all assets and no liabilities and the taxpayer would be fine.
It is actually kind of rude and annoying.
SR
Mmm, I don't think my personal IRS tax return data are subject to FOIA requests by random journalists. Why would my mortgage data be, if I had one?
I would think that there would at least be a requirement to anonymize the data before turning it over to random journalists. It's been a few years since I looked at privacy law, but the gov't has greater obligation to protect personal data then anyone does in the private sector. Is the new gov't owned gse sector public enough to trigger the higher level of privacy protection? I sort of hope so.
If nothing else, such freedom of access to records is a bloody invitation to identity theft.
I am saying that this blog suffers from MASSIVE group think on certain posts by Tanta.
Insofar as that's true, accusing the groupthinkers of being a cult is far less likely to produce a change than countering the arguments does, and much more likely that any cogent counterarguments will be buried in venomous and uninformative exchanges. There are a number of posters above discussing how they could use the kind of data GM was (possibly sarcastically) suggesting be available. Do you think the cultist charges and resulting defenses make it more or less likely a casual browser will get to those arguments?
Ann, F.E. - I think the first part of the article was tongue in cheek, but the latter part, where she discussed "that idea" was not.
Copout. You can't have the data because you can't handle the data? Right....
Sorry, there are many ways to provide the data that can easily be handled by 'PCs' and rather simply too.
She's obviously asking that question because she doesn't trust the answers we are getting or have gotten from this institution which we the people now own. Though I'm sure she as everyone else knows that they already have detailed stats on everything. That's how they kept making such "lucrative" business decisions.
I am not here to change minds. That is a losing proposition.
Just remember when you are gathering infomation you are only as good as the opinions around you. I am highlighting that certain opinions are biased and out of touch. That is all.
The CULT references simple make it easier for people to understand what I am saying and enjoy readind the post.
SR
part of the problem in analyzing 65,000,000 files is that you don't have permission to invade the privacy of any of the people behind those files. Sure, some numbers may look fishy but until you examine the original loan documents and verify the original claims made during the loan application process, you cannot make any useful conclusions about the propriety of any one loan. What was lacking then and is still lacking is sufficient numbers of skilled loan officers making good judgements. The rush to profits on the part of senior management overrode any fuss that came out of the back office.
The typos hurt the above intended effect.
SR
Please stop with the stupid privacy arguments. Just take all personal information out. Really you can't say they already provide this info on one hand and then on the other say if they provide it there will be massive invasion of privacy. It's just stupid.
All,
The first comment (Stalky) had it right. PostgreSQL could handle the data as could MySQL.
At 1 KB per mortgage, it would take around 12 DVDs to store the data. At 1 MB per second, the data could be downloaded in less than one day.
Could most members of the public handle a project like this. No. However, lots of folks on this list could.
I have to agree with those saying that this is a silly blog entry; yes, lots of us could do useful things with 63 million or so mortgages, and one could fit at least a page of info on each mortgage on a single BDROM. Granted, making available high-resolution copies of 62e+6 original loan documents packages would be dumb, but they wouldn't be analyzable anyways. I'd be surprised if there was more than a few terabytes of analyzable data at all.
As a large-scale data modeler (please, hold the boos), I would LOVE to slice-and-dice the 65M raw data records. Summary data can hide a lot of sins.
As for the suggestion of "conversion" to PDF format, it reminds me of a common technique used by scientists when forced to publish data they really don't want to publish: convert complicated tables to PDF, preferably rotated and broken across multiple pages to make scanning back into a useful numeric format cruelly difficult. Sometimes summer interns are hired to re-type the tables as part of their purported scientific "education"... oy veh...
Actually surely the suggestion was figurative and not literal? Someone with fiduciary responsibility vis-a-vis taxpayers, e.g. Treasury accountants and financial analysts, should over the coming weeks vet the books/balance sheets of FNM and FRE.
Well, THIS taxpayer has terabytes of storage and a cluster of over 150 quad-processor servers at his fingertips. I'd love to crunch on that data myself.
Loan Performance which covers a good bit of subprime, Alt A and Whole Loan Prime, covers about 25 million loans, with monthly updates going upwards of a billion rows. With minimal indexing, it's about 1.5 terabytes. This is within the realm of what you can by at Circuit City but any interesting queries are going to take days.
Um, Tanta, I have a great deal of respect for your work, but I think there are better uses for your tremendous talents than mocking reasonable proposals. Flooding the dark rooms of Fannie and Freddie with some sunlight isn't such a bad idea, though I don't know what it would accomplish in the end.
Last I checked, when there is a large disclosure like the proposal, the press and the blogosphere will get copies, put them in real SQL databases, and do the analysis. It's not hard. Then the press and the blogosphere report on what they find to the rest of the world. This doesn't seem worth your time to ridicule.
Downloading, dissecting and giving painstaking and scrutinizing analysis to the numbers and doc's may not be helpful to determine the magnitude of fraud and the thousands of culpable parties at every stage of manufacturing these trillions.
What is significant is that the extent of the unraveling values of assets and liabilities proceeds at and ever faster rate with only bottom guessers suggesting an end in sight.
New thread? Please?
Tanta is right. This is a major project that I would be reluctant to tackle unless I had absolutely no choice (and I have handled some HUGE e-discovery projects in my short legal career even if you do not count my number crunching for various political campaigns). There comes a point when you can have too much data and this would easily be one of those times.
"For example, I didn't stop to consider Tanta's lame news judgment in choosing now as the time to protest the conductor's selection of "Nearer My God to Thee." Glub, glub..."
lol
Taking Peter Schaeffer estimate of 12 DVD we arrive at about 54GB total. It isn't going to fit on a BD-ROM (as per Jwhitland). But it is easily doable with BitTorrent.
Somebody with access to that data please remove the personally-identifiable information and post in on Pirate Bay. They already have over 100 torrents that are this big or bigger.
Bring it on!
To answer the post-mortems: the original loan doc's and ALL THE SUPPORTING DOC's must be compared and confirmed with veracity. Crunching spread sheets won't get you there.
That's why evolution demanded professional independent auditing. Thanks, SarbOx. It's tedious and costly, but failing to do so is more expensive. Nature is replete with tedium... by which most stable and solid growth occurs across every realm: phyical, biological, empirical and human endeavors.
Good luck without paying attention to the details. See how well that's turning out?
s-r, you've posted 19 of the first 130 comments. Repetition isn't the most effective form of argument.
So how are Billy Joe and Bobby Sue Taxpayer going to examine a database of 62.5 million loans? - Tanta
62 million loans to check out? I bet Mortgage Pig could do it and over night to boot. Bobby & Sue can still trust Mortgage Pig even if they can't trust Wall Street...
I love you Tanta (I really do), but like:
l.a.guy: HaloScan.com - Comments
and Fair Economist: HaloScan.com - Comments
I think you are finding fault in the details rather than the goal. Taxpayers (or their representatives) deserve to know exactly (and I mean exactly, not in summary) how deep the doo doo is since we and our children will be cleaning up this mess.
Also, on a different note, I suggest you research team-building and personality types.
I worked a company with huge conflicts among the management team. The company shrank to a remnant of its former self. Apparently, the company that was handling our payroll (Kimstaff?) offered some team-building seminar. When we got the part about identifying personality types things became clear.
For example, strong leadership/entrepreneur/action-oriented types really dislike talking to the detail-oriented/conscientious/analytical types because they want straight info in brief, right now, and usually that is at odds with someone who wants to deliver complete and accurate information with dependencies and qualifiers.
Likewise, the detail-oriented/conscientious/analytical types detest listening to the social/persuasive/sales-and-marketing types because those types prefer persuasion over information and unfortunately are very successful at it.
Anyway, I think this might explain your disatisfaction with GM. Not to put you in a box, but your posts clearly indicate a desire to set straight the information (which often appears at odds with how GM is trying to sell a point).
Of course, I have to point out I'm speaking from the memory. I am a software engineer, not a human-resources or psycholy professional. So, I should be corrected by more knowledgeable experts if I'm in error about this.
After our company experienced the team-building seminar, there was no memo to make nice and be friendly, no incentive to get along, yet I believe the people in our workplace naturally came to accept the group dynamic once it was understood there was nothing personal in these cross-personality transactions, and that you really do want (for example) one of these social/persuasive type people interacting with the public and the clients/buyers/investors.
In an ideal world, GM would be have brains AND beauty so more of us would tune in and be accurately informed. But let's face it. Nobody has it all, not GM, not Einstein, not Obama.
I am thankful there is somebody like you to educate those of us outside the industry about all the uber-geek mortgage goodness.
Also, as someone familiar with current home computer technology, I would differ with the IT geeks who say 62.5 million records can be managed no problem on a home computer.
Say the address, price, interest rate, and other mortgage data could be stored in just 40 bytes per record. For comparison, this single sentence (uncompressed) would require 147 bytes of storage (including the period and a null terminator character).
Anyway, 40 bytes * 62,500,000 entries = 250,000,000 bytes.
There are 1,073,741,824 bytes in a Gibibyte so our 250,000,000 bytes are also known as 2.39 GiB (aka GB or G). My machine here at work has 2 GB of physical memory.
Microsoft Windows XP manages memory in a "virtual" fashion, so for example, it is transparent to me (as a user) that the OS does not actually have the entire file in memory all at the same time (part of it is being stored off on disc), however, it is immediately obvious to me whenever I exceed this "magic" threshold of physical memory on my machine because everything slows to a crawl as the machine must read and write out tons of memory in and out to and from the hard disc.
So, it is fair to say data of this magnitude would be unwieldy on a personal computer. Okay IT guys, go ahead and have at it. Maybe I missed something or miscalculated.
stock-regulator
What interesting questions are you going to ask the data with all the personal information stripped out?
We already know that many but not all stated incomes were false. We already know many but not all appraisals were inflated. That is why Fannie went from a profit of several thousand million in 2006 to a loss of a few thousand million in 2008.
Without the ability to verify the data in individual loans, what else can you find out?
Fanny and Freddie only have conforming loans, right? Isn't that how they are defined?
Doesn't seem possible they have ANY subprime.
That, too me anyway, was Morgenson biggest gaff. And you didn't even mention it!!
Is ignorance feeding fear, perhaps?
Zuzu,
For you, dearie...
Aubade (Philip Larkin)
I work all day, and get half-drunk at night.
Waking at four to soundless dark, I stare.
In time the curtain-edges will grow light.
Till then I see what's really always there:
Unresting death, a whole day nearer now,
Making all thought impossible but how
And where and when I shall myself die.
Arid interrogation: yet the dread
Of dying, and being dead,
Flashes afresh to hold and horrify.
The mind blanks at the glare. Not in remorse
- The good not done, the love not given, time
Torn off unused - nor wretchedly because
An only life can take so long to climb
Clear of its wrong beginnings, and may never;
But at the total emptiness for ever,
The sure extinction that we travel to
And shall be lost in always. Not to be here,
Not to be anywhere,
And soon; nothing more terrible, nothing more true.
This is a special way of being afraid
No trick dispels. Religion used to try,
That vast, moth-eaten musical brocade
Created to pretend we never die,
And specious stuff that says No rational being
Can fear a thing it will not feel, not seeing
That this is what we fear - no sight, no sound,
No touch or taste or smell, nothing to think with,
Nothing to love or link with,
The anasthetic from which none come round.
And so it stays just on the edge of vision,
A small, unfocused blur, a standing chill
That slows each impulse down to indecision.
Most things may never happen: this one will,
And realisation of it rages out
In furnace-fear when we are caught without
People or drink. Courage is no good:
It means not scaring others. Being brave
Lets no one off the grave.
Death is no different whined at than withstood.
Slowly light strengthens, and the room takes shape.
It stands plain as a wardrobe, what we know,
Have always known, know that we can't escape,
Yet can't accept. One side will have to go.
Meanwhile telephones crouch, getting ready to ring
In locked-up offices, and all the uncaring
Intricate rented world begins to rouse.
The sky is white as clay, with no sun.
Work has to be done.
Postmen like doctors go from house to house.
Tanta,
On this one you seem to have strayed from your area of expertise, but that's good because it's a great opportunity to highlight just how technically feasible Morgenson's proposal is.
As a starting point, have a look at Amazon's Elastic Computing and Simple Storage web services.
Please do scroll down and look at their prices.
Computing power, storage, bandwidth are not remotely the problems here.
What is likely to keep this from happening is Congressional nonfeasance (props to Barry Ritholtz for that term) and REIT lobbying, including Fannie/Freddie's.
(on that last one, if anyone believes that the recent law about them not being allowed to lobby is going to be respected, please contact me for some AWESOME bridge opportunities!)
If, by some miracle, this data did become available, the crucial ingredient to making it useful would not be CPU cycles and database tables. It would be - cough, cough - the willingness of people with domain knowledge and an honest agenda to point the database nerds in the right direction.
Game?
Grrrr...
My figure of 250,000,000 bytes above should read 2,500,000,000 bytes.
My answer, based upon a hypothetical record size of 40 bytes per mortgage data record, is still 2.39 GB.
@Lurking Larry: either you are using bad software or you are just bad programmer.
With sensible software the AP (analytical processing) queries essentially run at the speed of streamed reading from the hard drive.
We aren't talking here about OLAP or SQL. The data set is static, so there's no use of any advanced database technology or indexing. Just plow through the file record by record and as many times as needed.
However clearly you are pointing to the fact that many people may not be aware of the proper methods to process static data sets or simply not familiar with required software tools. And here I fully agree with you. But the problem isn't technical, it's the problem of personnel and/or education.
@other jim:> What interesting questions are you going to ask the data with all the personal information stripped out?
This is a whole field of knowledge. Please google two things:
1) privacy preserving data mining
2) personally identifiable statistical queries linear programming
"With sensible software the AP (analytical processing) queries essentially run at the speed of streamed reading from the hard drive."
Maybe it's time to move this thread to <a href="http://slashdot.org/>Slashdot?
I could care less about arguing the practical details of this particular post. I follow Tanta for her wit and style and curmudgeonly disposition. Good writers don't have to be correct to be enjoyable. That's why no matter how correct certain of you may be, (and I'm not conceding any of you are) you'll never be able to hold a candle to Tanta's prose. Your writing is offensive and boorish - regardless of its technical merit.
One problem with Gretchen is that she's too saturated in sweetness and lightness to handle serious subjects. Her style is weak. When she tries to be tough it comes across as silly because its unnatural. Most of the NY Times suffers from this.
Sylvester, my point was to suggest a typical personal computer will not run very fast browsing a 2.39 GB Excel file or Access database.
Perhaps there are readers here who have operated on files this size and can correct me.
Freddie Mac already does a loan level disclosure. Do the research Gretchen!
uber nerds of a different type came out on this one
Freddie Mac already does a loan level disclosure. Do the research Gretchen!
Freddie Mac Announces Monthly Loan-Level Disclosure Updates to Mortgage PC Securities. - News Archive - Freddie Mac
Maybe Microsoft could have taken that $300M they spent on those clever ads with Seinfeld and changed the Access file limit size.
Maybe they're waiting for a competitor to do it first so MSFT can just copy it.
Boo on you, Tanta. And the "tranches" actually ARE the loans, too--or part of them. The "tranches" can't be rated without knowing if the houses are there, or are habitable, or whatever. So there!
why not have the database online and let one query with a web front-end the same way access to the patent/trademark database is handled? those that want access to the raw data can purchase a dvd rom for cost and import into whatever database they wish. mssql, postgress, oracle and countless other dbs can easily query 100M+ records
tanta is getting tiresome. I simply skip most of her posts. As an IT professional, I would love to have that data, to form my own conclusions.
CR, may be you need to make some tough personnel decisions here.
@Lurking Larry: And my point was that you don't want to cripple yourself by using Excel or Access.
US government routinely sells the data of similar nature and quantity to the private industry. They use very basic fixed field fixed record-length text files. Typically there are about 928 files in such a data set, one file for each SCF (first 3 digits of zip code). Due to extremely repetitive nature of those files they compress very well using standard compression utilities like PKZIP or WinZIP.
Grab an example from USPS here: http://ribbs.usps.gov/files/Addressing/PUBS/AIS.PDF
The tools to analize such files are available both for free and for money. But you have to google for them a little deeper than you would for Excel or Access.
Again, the problem is not technical in nature.
Gretchen's suggestion makes plenty of sense to me, Tanta. Obviously she is thinking that even if the average taxpayer won't examine the records, commentators such as reporters or bloggers with a budget could do so. Probably there are valuable lessons to be learned. Sunlight is the best disinfectant.
Fannie and Freddie both offer detailed financials - they show first profits and losses, and while I did not review in past, I don't recall massive losses
Second - the current financials show portfolio breakdown and quality along woth performance by segment - and in Q1 and Q2 2008 - when you take out the fair market writedowns - there were not huge losses
In fact - from memory - if I recall my review Fannie, if you use only the liquid/cash reserves (not the tax credits on future earnings) and added 25% of so to the existing losses - they could go years before running out of money
If alleged experts like Gretchen want to review performance they could and SHOULD start with reviewing the detailed existing available financial information and data - and presenting an intelligent meaningful real world review of the financial strength or weaknesses
It is my understanding neither Fannie or Freddie ever had significant investment, exposure or losses in subprime. The currently have comparatively no exposure to subprime risk.
They also have a limited amount of ALT-A exposure - just over 10% of Fannie's current book if I recall. And they have all but stopped adding ALT-A as well
So Fannies current picture - and portfolio - seems it represents the worst case exposure for them
I think Tanta is correct ... these alleged experts in the media refuse to do any intelligent detailed review on existing available data and information - why would they do anything different with even more data and info?
And quite honestly what does it matter? The public ally reported data splits out by loan type - what benefit is there to go to a loan level revile?
Especially when if I recall correctly Fannie has never had huge losses in subprime etc in pasty or to date ...
Again Tanta is right here - the MSM refuses to make even a nominal effort it seems to do even the most rudimentary review of the financial - of facts - for any of these situations
Had they, in my opinion, we would all be better informed and if, as I believe, the data shows things to be far better than the perception, we might not be in such a death spiral right now ...
The point is NOT that the average person would know what to do with the data. The point is that experts could analyze the data openly, claims could be verified, and inconsistencies brought to light.
It's the same way that science works. You have to publish your results and data openly so it can undergo peer review. I'd like F&F to be peer reviewed.
And by the way, all the requested data could fit into 8 bytes per loan, for a total file size of about 500 MB for all the loan data. That's a very reasonable file size, amenable to analysis.
The moral content of economic terminology in the popular press: A guide
Foreign direct investment (inbound): Good
Current account deficit: Bad
Trade deficit: Catastrophic
Capital account deficit: Not used. Presumably bad.
Weak national currency: Bad
Exports: Good
Imports: Bad
Rising house prices: Good
Falling house prices: Bad
Affordable houses: Good
Unaffordable houses: Bad
Free trade: Neutral
Unfettered free trade: Bad
Fair trade: Good
Outsourcing: Evil
Buying local produce: Divine
Pay rises: Good
Low interest rates: Good
Inflation: Bad
Communism: Very Bad
Socialism: Bad
Capitalism: Bad
Bluematter.: The moral content of economic terminology in the popular press: A guide
Unfortunately the one who "dosen't get it here" isn't Gretchen. Modern databases, like the ones used at Freddie and Fannie, can plow through 62.5 million "rows" (if that's the limit of your understanding of data) in no time.
I concur what @reason wrote: "for a total file size of about 500 MB" This is very true. The data we talk about is extremely repetitive. If instead of using simple WinZIP we used one of many specialized compression tools (e.g. based on PATRICIA-trees) the data would fit on the CD-ROM, not even a DVD would be reqiuired.
The commercial tools to analyze such data sets were available since the CD-burners became popular and ran efficiently on the typical office computers from the same era. All that predates even the use of DVD and DVD-burners.
Again, technology is not a problem. The knowledge is (or rather lack of it.)
renter writes:
tanta is getting tiresome. I simply skip most of her posts. As an IT professional, I would love to have that data, to form my own conclusions.
What is getting tiresome are people who would rather kick the scapegoat rather than analyze the numbers.
Get the data yourself.
Freddie Mac already does a loan level disclosure.
Freddie Mac News Center a..._loanlevel.html
How about in a CSV file that can be data mined with SAS?
How about a CSV also with the list of names and current addresses those that have walked away from their obligations?
You can buy gse loan level data from huduser.org
The fields may not be exactly what Grethchen wants but it is loan level data.
Data Sets - HUD USER - Policy Development and Research's Information Service
I'm coming in very late on this, but I think what GM is arguing is that one cannot trust the "summary" statements we've been seeing for the past few years to be truthful. Assuming that's true (and, to state my own bias, I think that's clearly been the case), what's the solution other than "full disclosure"? Granted, I would not slog through a hundred million datasets, but somebody would, preferably somebody without a direct financial interest in making it appear that an entity's numbers look awesome. Such analysis might have (I would actually argue would have, but I try to dial back my own enthusiasm) prevented the ongoing plunge from the stratosphere we're currently witnessing...
ever pays to look just at the paper; that's how things got so bad. Look at the actual thing. Go out and look at the houses. "The map is not the territory."--Alford Korzybski
While I normally hang on Tanta's every word, I'm with G.M. on this one too. Now that Frannie are firmly under the government's wing especially, more transparency is better. There is a market for the information and technically it's not intractable. Get it out there in a format people can really use, and folks will start to use it, and those who do will do value-added analysis that will be worth a lot of money to taxpayers and investors.
More importantly, now that we've given this administration and its successor the ability to fiddle with Frannie, there's a high likelihood we'll regret not having that transparency, because it'll breed scandals...
the willingness of people with domain knowledge and an honest agenda to point the database nerds in the right direction.
What of people who are both database nerds and replete with domain knowledge? Sasquatchian? We're slaves to 'da man'?
Or we're robots who don't pay any taxes? We're just part of a vast conspiracy? Riiiiiiiiiiiiiiiight.
Yep, I agree with GM and sotck regulator, we need that data. I'm sure many wonderful things can be done with it. I myself can't figure out any use to me personally, but with that much data surely we can prove or disprove the existence of God. It's the mortgage equivalence of the hadron collider. Lots and lots of data for the conspiracy minded and something for the IT guys to do on a Saturday night.
I can't seem to do anything with the information I've got, but with more data points I will be able to prove, beyond a shadow of a doubt, that large corporations aren't always run at optimum efficiency and that politicians are swayed by money. WE NEED TO PROVE THIS, STAT!
something for the IT guys to do on a Saturday night.
that would involve them giving up other pursuits, you know.
I love it when people fill their comments with lyrics and poems written by other people.
It's so ... interesting.
Arun,
Nice poems.
Lisa,
aLL i CAN say is "Do anybody make real SHIT anymore?"
Though not explicitly stated, I assume she's speaking about the retained portfolio. Your claims about the inability to account for those loans because of size are incorrect. Freddie has been posting the details of all MBS loans issued since late 2005, a lot of data. There are systems that deal in terabytes that easily could handle such a project. I know because I work with them. Have Gretchen give me a call.
FIRST
Sylvester,
I downloaded the AIS .pdf file in your link. I think your point is the government routinely delivers large databases to its clients.
Sorry I didn't point this out before, but it appears you and others have focused on the statement Tanta made about delivery of information on magnetic tape (which, by the way, I believe was a snarky comment and not a guess about a future government distribution plan). I never said the data could not be compressed, or that it couldn't be delivered on CD, DVD, or torrent. I agree it could.
My posts have focused on affirming the limitations Tanta suggested about using Excel or Access to examine such data. For example, my estimate of the uncompressed database is 2.39 GB and Tanta mentioned some versions of Access can not support single files beyond 2 GB.
So, in defense of Tanta, it seems unlikely Billy Joe and Bobby Sue Taxpayer are going to examine a database of 62.5 million loans using Excel and Access. But this is moot anyway, because I do not believe Billy Joe and Bobby Sue need to examine the database themselves so long as someone without an agenda does (and gets the word out.)
I would love to have loan-level time-of-origination data for the GSEs, but even that's not enough. I'd kill for monthly delinquency status on those loans, like that provided in LoanPerformance.
Its clear she didn't think this through. But I am reminded here of an English Professor who is criticizing the Grammar of the proof of Fermat's last theorem. She is requesting more transparency. How about a simple well done random sample of a class of loans made. Complete detail on the sample pulled double blind from the database. Whats the big dieal?