Cheering for Open Hearings in San Jose

The Merc issued a smart editorial today demanding that arbitration hearings that affect taxpayer money should be opened to the public.

I cheered them on, with the below statement.  Enjoy.

Access to the public is exactly appropriate for any arbitration hearing that affects the public's money. We should go further, however, and demand that all negotiations that affect public funds, such as those highlighted by Councilman Oliverio, should be recorded and made public after a suitable embargo period. This embargo would allow negotiators enough latitude to propose compromises, while remaining eventually accountable to their constituents–whether union members, shareholders, or voters. Even India, with some of the worst corruption in the world, has implemented their Right to Information act, putting similar information into the public domain after 30 days. Democracy only works when our right to know is respected. It is ridiculous that our access to information is worse than India's. Kohl S. Gill Sunnyvale

Scientists Playing with FOIA, Bound to Get Burned: Stop Worrying, Embrace Transparency

Today's scientists are effectively public servants, and should start behaving accordingly.  The public has a right to know what scientists do and how they do it.  To prevent scandals like Climategate, scientific correspondence, critiques, and even data manipulation must be done in public.  The University of East Anglia and associated climate scientists didn't understand that. I'm not sure they do, even now that their emails were hacked and used to sow doubt about climate science.  Jon Stewart sums up the problem better than I can:

The Daily Show With Jon Stewart	Mon - Thurs 11p / 10c
Scientists Hide Global Warming Data
www.thedailyshow.com
Daily Show Full Episodes Political Humor Health Care Crisis

This isn't an excuse for climate change denialists.  (Please don't call those folks "skeptics".)  Climate change  is real, it's caused by man, and there are reasonable steps we can take to halt and reverse the damage.

The hackers who extracted emails from UEA committed a crime.  In so doing, they exposed that several scientists called people dismissive names, manipulated access to blogs, adjusted data in non-obvious ways, and derided their obligations under the law such as the U.S. and U.K. Freedom of Information Acts (FOIAs).


What bothered me most was the disregard for the law.  The most damning example from the emails I've found includes this passage, from Phil Jones to Michael Mann (emphasis mine):

Mike,

I presume congratulations are in order - so congrats etc !

Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it - thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who'll say we must adhere to it !

The passage goes on to other topics, but you get the picture.

The reaction to this crime should not be bigger and thicker firewalls, or at least not solely that.  Scientists should realize by now that privacy is not absolute, and in most cases, it's not even a good idea.  They should embrace transparency, by moving correspondence to more open formats like blogs, wikis, and Google Wave.  This move is critical to maintaining the long-term respect and public support for science.  Obviously these platforms need to be adjusted to accommodate the complex manipulations of large data sets; the scientific community is in the best position to demand for and contribute to the development of appropriate applications.  They will find great allies in the library science and open government communities, folks tackling the same essential problem.  One great example is the Alliance for Taxpayer Access.  In any case, scientists need to get out in front regarding transparency, or risk letting deniers define the route, themselves.

Some scientists may argue at this point, claiming that such presumptions of transparency will stifle free and open discussion and debate, that scientists working in the public sphere will censor themselves, and that the best ideas will not come forward.  They might worry that science would become even more politicized than it already is.  These might be fair points; after all, much of scientific inquiry is based on proposing and discarding ideas, ideas that seem crazy at first, but may just be right.

The answer is to embrace the embargo.  Embargoes are already used throughout the scientific enterprise.  Usually, they're used by scientific journals to delay access to non-paying readers, but they can be used in other ways, as well.  Scientist A collects data and wants to analyze it and publish.  While working on it, she shares her data with Scientist B so as not to delay further analysis.  Oftentimes, as a courtesy or by agreement, Scientist B holds his own results under an embargo until Scientist A has had a chance to publish her results.  This avoids confusion over which person should get credit for the initial data and analysis.

Wary scientists should know that everything they do, every email they write, every correction of data, every keystroke, could eventually wind up in the public domain.  The platforms that will serve those scientists best will incorporate a time-bound embargo, with definite and obvious, rolling expiration dates.

As Judith Curry of Georgia Tech puts it (emphasis mine):

[G]iven the growing policy relevance of climate data, increasingly higher standards must be applied to the transparency and availability of climate data and metadata. These standards should be clarified, applied and enforced by the relevant national funding agencies and professional societies that publish scientific journals... The need for public credibility and transparency has dramatically increased in recent years as the policy relevance of climate research has increased. The climate research enterprise has not yet adapted to this need, and our institutions need to strategize to respond to this need.

This actually isn't a loss of privacy–which doesn't really exist, anyway–but rather a move to make science even more legitimate and accessible to the public.  We should all recognize the great value the world has derived from access to earlier scientific correspondence.  Occasionally, the public needs to be reminded that scientific inquiry is a human enterprise.  Transparent scientists are ones they can believe in.

[P.S.: Judith Curry has done a fantastic job of corresponding with (initially hostile) commenters at the Climate Audit blog.  Some of her top comments are here, here, here, here, here, here, here, here, here, here, here, here, here and here.]


[Edit: Added emphasis.]

Yahoo Thinks You're Gay. Now, Find Out Why.

Ever wonder why you get the particular advertisements you see when using tools like Yahoo?  Most of the time, the ads relate to the content you're accessing at the time, like ads for pet services when you spend all day looking at cute pictures of kittens instead of applying for jobs like you promised you would.

But even if you're not logged in at sites like Yahoo!, Google, or Facebook, those companies tend to collect quite a bit of information on you, creating their own consumer profile of you, which they use to match you up with targeted ads.  Wouldn't it be great to take a look at that profile?  Wouldn't you like the option to edit such a profile, or turn it off altogether?

Thanks to increasing pressure from transparency advocates and the U.S. Federal Trade Commission, Yahoo! is taking a baby step in this direction.  Yahoo! is launching a test version of it's Ad Interest Manager (AIM), which shows you a summary of the info that Yahoo! has on you.  I haven't logged in at Yahoo! in a while, so without logging in, I took this screenshot of the AIM site.  Now, it definitely gets some things right, like my age range, gender, OS and browser.  It still thinks I'm in DC, which is odd because I'm on the other coast, now.  When I log in, I see exactly the same profile, except that "Entertainment" has been added as an interest.

Note that Yahoo! appears to know who I am, even when I'm logged out!  Yahoo's policy is to de-identify data after 6 months at the most, but they never delete it.

To be sure, this kind of access is only the beginning of what consumers should demand, and the AIM tool has received mixed reviews from confusing, but better than Google to half-hearted.  But, it is a start.

Are there alternative search engines that will show you what data they collect?

[Google's closest analog - Dashboard]

[Edit: Grammar, and the last question.]

Lies, Damn Lies, and Chinese Math: Over 1 Billion Snowed

Even the Chinese don't trust Chinese statistics, and they have the catchy phrases to prove it.

Apparently Science Magazine's Letters section is the only venue to pick up the Xinhua News Agency article on the new regulation issued by the Chinese National Bureau of Statistics, which clarifies the penalties and procedures for prosecuting statistical corruption.

The letter, by Junguo Liu and Hong Yang of Beijing Forestry University and Swiss Federal Institute of Aquatic Science and Technology, respectively, gives an intriguing insight into how the Chinese people view the corruption in their own statistics:

Statistical corruption has been found in China for years, largely for two reasons. First, economic growth is a key factor determining the promotion of government officials. Statistical data and numbers are regarded as a reflection of economic growth, which is used to evaluate the performance of the officials. This is the so-called "numbers make leaders" phenomenon ("shu zi chu guan" in Chinese). Second, the statistical organizations are not independent entities in China. They are a part of the government and hence are vulnerable to government interference. Without specific laws and regulations to punish statistical corruption, government leaders can intervene in statistical reporting with low political risks. They may tailor statistics for different purposes, such as inflating statistical numbers that indicate economic achievements and decreasing statistical numbers for environmental pollution and damage. This is the so-called "leaders make numbers" phenomenon ("guan chu shu zi" in Chinese).

My guess is that the move by the NBS to curb official corruption is at least partly due to the recent scandal surrounding Qiu Xiaohua, former director of the National Bureau of Statistics, who was sacked and expelled from the Chinese Communist Party in 2007.

Jefferson, Interrupted

Thomas Jefferson's words are heavily censored in his own memorial in Washington, DC.  Consider this passage from the Declaration of Independence, inscribed on the interior of the memorial:

We hold these truths to be self-evident that all men are created equal, that they are endowed by their Creator with certain inalienable rights, among these are life, liberty, and the pursuit of happiness, that to secure these rights governments are instituted among men. We...solemnly publish and declare, that these colonies are and of right ought to be free and independent states...And for the support of this declaration, with a firm reliance on the protection of divine providence, we mutually pledge our lives, our fortunes, and our sacred honour.

Despite the fact that the passage in yellow was never written by Jefferson, this looks fair enough, until you take a look at the actual text of the Declaration:

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness.

The passage in green, which Jefferson thought was the whole point of the Declaration, was dropped.  Some editing to conserve space is appropriate, but this gutting of his message is ridiculous.  Of course, I'm not the first to notice this censorship.  Let's hope I won't be the last.

In light of Mr. Jefferson's prominent role as the supplier of our new lead quotation here, let's raise a glass to this Founding Father, censored in his own memory.

Update: Since folks have asked, my point is that this type of editing is dangerously Orwellian.  Even our national monuments may be lying to us.  (The memorial was constructed just a few years before Orwell completed 1984.)

(Thanks to Tony the Misfit for the gorgeous image of the Memorial.)

What's Wrong with Privacy? (Transparency Matters #1)

Over the past few years, I've become something of a transparency nut. One of the problems with transparency as movement is that very few people even recognize the term, much less know what it is. Once they see an example of transparency, they're often quite supportive.

I'm taking a different tack. I'm going to outline why privacy is bad for you. Bad for you, bad for your health, bad for your family, bad for your economy, bad for your politics, bad for your security, bad for your planet.

Or, at least I'm going to try.

Privacy as a social phenomenon is not terribly old. Indeed, the first taste that most Americans experienced was the privacy of isolation, and then of anonymity, which they got by moving to the American wilderness, and then to cities and suburbs.

Privacy as an activity is not terribly natural. Privacy isn't holy, or even well-defined as a philosophical or legal concept. The US Constitution doesn't explicitly lay out a right to privacy; the First, Fourth, Ninth, and Fourteeth Amendments are often stretched to serve as such. The courts don't recognize privacy except in those cases where citizens have a reasonable expectation of privacy.

Privacy is inconvenient. As we see in the social networking phenomenon and online advertising, folks are trading more information about themselves and their habits in exchange for better-tailored goods and services. Though most social networks, e.g., Facebook, have detailed privacy settings, most users leave these settings off for convenience.

"You have zero privacy anyway. Get over it." - Scott McNealy, Sun Microsystems, 1999

Privacy is illusory. Private detectives agree that the most effective way to get information about someone is not through cameras, microphones, wiretaps, or keystroke monitoring. Rather, it's much easier to find all you need to know by using Google, sifting through publicly accessible civil and divorce court records, criminal and driving records, credit histories, physical mail, digging into garbage, and chatting up that gossipy neighbor or brother-in-law.

Privacy is distracting. Privacy is used as a solution to more fundamental social concerns. You shouldn't have to keep your information private to prevent harassment, identity theft, or stalking. Those are already crimes, in and of themselves, and privacy is a flimsy barrier to them, at best.

Privacy is corrosive. Many social movements have their roots in liberating folks who do not fit the stereotype of normal, and bringing the falsity of those stereotypes out into daylight. A short list of relevant issues would include: domestic physical and sexual abuse (privacy of the family); homosexuality (privacy of sexual behavior); as well as alcoholism, drug abuse, dyslexia, AIDS (privacy of medical conditions). In each of these cases, the first step toward dealing with the issue was and is confronting the damage that privacy has done.

Privacy is contagious. Modern corporations and other firms have taken the idea of privacy into the business sphere and we continue to see the repercussions. Untraceable banking transactions are only one example of a practice that facilitates more than just the recent financial crisis. Consider drug-running, gun- and ammunition-smuggling, human trafficking, and terrorist criminal enterprises. If you could trace every dollar that changes hands, you would be a long way toward preventing and prosecuting crimes as well as protecting and providing restitution for victims. If you could open the books of every firm, financial crises like this could be avoided.

Privacy is expensive. Privacy is now a huge industry. Privacy is not insurance, which has the value of increasing your resilience in a measurable way. Rather, privacy purports to increase your protection, with few guarantees. You trade your information, often along with your money, to Company A for goods and services. Then you pay Company B to make sure that Company A doesn't release that same information. And only rarely do you get the opportunity to see the information that has been collected about you – to see how it is being used, for example – even to simply correct the information itself.

Privacy is wrong. You know that your government works for you, but you still see excessive secrecy in all the functions of government. Scandals involving abuse of power, corruption, waste and fraud abound as a direct result of privacy. This privacy has crept into government decision-making, awarding of contracts, evaluation of contractors and the operations of those contractors. You should be able to access any information involving the government or its functions without encumbrance or delay. The only legitimate delay is when the privacy involved is crucial to the security of the nation.

Privacy is insidious. You want to be paid well, and to be paid fairly, for what you do, but you're often told to keep your wages private. This privacy allows you to be paid unfairly, by allowing arbitrary decisions, such as those based on gender, race, age, or hairstyle, to affect your wage. Also, since everyone is quiet about wages, you never know the true value of your work, and are never able to bargain based on that value. This has the net effect of suppressing the wages of truly valuable workers. This stems from the idea that evaluations, especially of individuals, are more accurate if they are private. Private evaluations are a sign of ineffective management. And making them public can even be fun.

Privacy is inefficient. You have to waste time, and you even receive lower-quality of care, because of waste in the medical industry. This waste is due to privacy considerations around the sharing of your medical records, at least in part. This privacy is supported by an absurd system that requires you to keep secrets from your current and future insurance companies, at the risk of your own health. Worse yet, medicine itself suffers from this privacy because researchers cannot access your information, even to find out which medical procedures are most effective for people like you.

Privacy is not necessary. There is an alternative. You can choose differently.

You can choose to create a more transparent society. Imagine a society in which you can voluntarily release information, with the understanding that you can always see what is done with that information, how it's used, how it's changed, and what the benefits and detriments are. Imagine a society where your neighbors don't snoop on you, not because they can't, but just because it's rude and they have better things to do. Imagine a society where you can do good by working hard, paying your taxes, and serving your community with the assurance that you can see just how other people are doing those things, too. Imagine a society where your governments, businesses, and other organizations don't waste precious resources just because they can get away with it. It could be a society where we all dispense with financial privacy to strike a resounding blow against organized crime, world-wide.

At the very least, such a society could be quite interesting.

Is this a society worth striving for? If so, how do we get there from here?

The Hiatus Ends

After over a year of silence, I'll be writing in this space again. (For some of the reasons I've been out, see: New Job, and Possible Radio Silence.)

I'm excited to resume writing about transparency, and I have lots and lots in the hopper to share. I'm also looking for paying work in the Bay Area as I move to Sunnyvale, CA, in October. The job search is tough, but it does allow me some time to devote to writing.

As you watch this space, you'll see several interweaving series of posts, on my last couple of jobs in the federal government, on my progress in finding the next great job, on why transparency matters, on great sources of information, and a whole lot more.

Let's get started.