Do Not Leave Those Naked Pictures of Yourself on Github

One of the not-so-entertaining results of our civilization's obsession with computing and cash is the "fuck our customers we don't care" approach taken by the consumer electronics and computing industry towards such things as systems administration, backup administration and security administration and its impact on our lives.  Now we are all forced to take on these dreary sub-specialties or face one of many horrible fates that this technology mania has brought down upon us: the wiped disk, the non-working backup, the zombie computer used by Chinese or Russian spies, or worse, the "hactivist" holier-than-thou swine ready to exploit your assets to mine Bitcoin or some other juvenile and anti-social goal for their self-appointed crusade.   We are all now responsible for these and many other tasks and woe unto those of you who think you are above such things for then your sins as documented by your iPhone will appear on social media and there you will be, in full color, engaging in some drunken bisexual orgy as an undergraduate for everyone to see just as you are running for your first political office or other responsible position.

Be warned, if you wish to avoid this or some other horrible fate, there are a few hundred things you need to pay attention to at any one time, although that list is a moving target. You have to know enough to keep yourself out of trouble.  No one else will do this for you. 

Many of us use Github as a repository for source code for our projects and collaborations. In the past I have used it off and on, but these days I use it more or less 7/24. As part of your repository, one could keep security strings that give access to various other resources that exist out there, such as the Amazon cloud. A friend did just that and forgot about it. Although he certainly knew better a few years later he made that repository public (it was either that or delete it, he wasn't working on that particular idea anymore).

Well his repository contained security information for his cloud account on Amazon which he also wasn't actively using and some hackers grabbed it and ran up a bill in the many 10s of thousands of dollars per day. Amazon.com caught it nearly immediately and my friend will not be liable for most of this bill, hopefully not any of it.

My friend is beating himself up because of course he knew better. He does know better, by the way. Don't let this happen to you. He suggests reading the following discussion on these issues to learn how to keep passwords out of your Git repository.

http://programmers.stackexchange.com/questions/163506/how-does-one-handle-sensitive-data-when-using-github-and-heroku

Never forget it's a jungle out there and that, generally speaking, people are scum.

The Nigerian Concept of "Cake"

What is odd about the debate around the concept of “cultural diversity” is that our culture here in America is a melange of bits and pieces of other cultures, mostly but not entirely European, in which we seem to have picked one from column A and two from column B and then declare that we invented it. Regional cuisine, Southern music, the tradition of the Christmas tree, our notions of a fair and just war, and so forth are mash ups of other people's cultures made our own with a few misunderstood regional historical events and a little stuffing for the poor oppressed turkey or great turnip replacement, the pumpkin.

Today I want to advocate that we borrow another cultural concept that I think is very appealing and has a lot of applicability to this great country of ours and which I propose we make our own. This is the Nigerian concept of “cake”.

What a healthy looking birthday cake with depth of field!

Like many countries in the world and in sub-Saharan Africa in particular, there are great extremes in wealth with the vast majority being dirt poor and a very few who are quite rich. But Nigeria is also blessed (or cursed as may be) with great natural resources, such as oil. The wealth generated by this oil is vast and yet most Nigerians are still extremely poor, with no improvement in their lives, no new schools, hospitals, and so forth. So where does all the money go?

Well Nigerians have noticed that nearly all the money goes either to the foreign oil companies or to the offshore personal bank accounts of senior government officials and some of their aides. Very few people in Nigeria benefit from the oil outside the government. There is some employment of local workers, but not much, and they are not paid well. Most of the engineers and managers who work for the oil companies come from outside Nigeria and return to their homes in other countries when their assignment is done.

Not only do the people of Nigeria not see any of this money, in general the Government does not see it either. These offshore bank accounts for Nigerians may have tens or hundreds of millions of dollars, but those are the personal accounts of their government or military officials. Apparently, only a small amount of this money actually goes back to the country in the form of tax or other revenues that might be used on public works projects.

An industrious Nigerian with some cake

The Nigerians calls this vast wealth from oil “cake” as in chocolate cake, or a birthday cake. They are not looking for justice, or a government who is not corrupt and provides real services for their people, or any of those worthy concepts. They may desire those things, but they realize that there is very little chance of ever getting them in Nigeria. At least not the way things have been going for the last few centuries. What they would like is a small part of the cake that is being pumped out of the country.

Being forthright and straightforward people, the Nigerians have come up with a variety of ways to get small pieces of the cake which they figure is at least as much theirs as it is the people who are profiting by it. One way to do this is to get some friends together, rent a rusted-out old ship of some sort, a few hundred empty barrels, punch a hole in one of the pipelines that run alongside the rivers, and voila we have a recipe for instant cake. The oil companies come by in a day or two and patch the hole, leaving the spilled oil where it lies.  Of course the government and the oil companies complain bitterly about oil theft but just who is stealing from whom, I wonder.   In this, energetic and productive fashion, Nigerians are able to make a much better living than digging in the mud with a stick or swatting flies as their government would like them to, and so benefit from the Globalization we hear so much about.

This shows initiative and a positive attitude on the part of the Nigerians.  A true demonstration of the entrepreneurial ideal.  Rather than waiting for someone to come along and "give them a job", they get off their rear ends and collect a little crude oil on their own.   

Now in our country, our government officials also profit on managing the natural resources and economy for the benefit of their rich masters but their fees are much less. In America, the corruption is more discreet as well and are usually channeled through reelection campaign funds and so forth.  The rest of us work for slave wages and although there is a larger middle class than in Nigeria, it doesn't add up to much. And besides, our government is working to reduce the middle class and being quite successful at it. When the Dow average goes up, it is the tiny wealthy elite in this country that benefits. The idea that it is the “retirement funds” that benefit as well is very much an obvious lie from the last century.

This fine concept of cake would have to be adapted to our American way of life of course.  The carved turnip from Africa must become a carved giant gourd here in America, metaphorically speaking.  We all want a piece of cake, but just punching holes in oil pipelines may be appropriate for the colorful native peoples of Africa but would not do for those in the west with our sophistication and finely honed environmental sensibilities.  But I am sure Americans can run with this idea and suitably modify it in a way that does not compromise other values and allows us to deceive ourselves into thinking that we invented the idea.

I hope that Americans will not let foolish pride stand in the way of adopting interesting ideas from other cultures.  If the world can adopt "blue jeans" and "jazz" from America, surely we can benefit from their concept of "cake" and thank the Nigerians for showing the way to a more just and fair world.

US Govt Still Denies Secret Nazi Base in Antarctica Leads to Hollow Earth

When it comes to conspiracy theories, the Nazi's clearly won the war.  Whether it is the secret Nazi mission to the moon, the escape of Adolph and Eva to South America, or the CIA work on the anti-gravity drive reverse engineered from the flying saucer that the Nazi's found when it crashed in Germany, a conspiracy theory with National Socialists immediately has a certain cachet that the others just can not approach.   The Underground Tunnel people and the Chemtrails people must grit their teeth in envy whenever they see the other nutty-boy theories flash a swastika, a graphic design they could never hope to be able to approach or surpass.

But with so many conspiracy theories to choose from, many of them overlapping and/or apparently conflicting, how can we know which theory to lie awake all night worrying about? Can our conspiracy theories work together or must they always be at each other's throats, a zero-sum struggle between mutually exclusive plots to destroy freedom and force us all to get socialized health care?

At Global Wahrman we advocate a unification of many disparate conspiracy theories to incorporate a National Socialist subplot combined with the accusation that the US Government acting through its UFO-hating CIA is trying to hide the truth.  This technique has the promise of taking a run-of-the-mill conspiracy theory and transforming it into a truly rewarding conspiracy theory that covers all the angles.

What attributes would such a maximum conspiracy theory, an uberconspiracytherorem as they might say in German, have?   For my money, its the conspiracy theory that combines the romance of far away and unknown places such as the Antarctic, with the all-season glamour of National Socialism, the hidden in plain sight Hollow Earth, inhabited by incredibly wise and powerful alien races, all hidden by a cover-up by a venal and power-hungry secret society that has taken over the government. The icing on the cake will be some sort of formal, academic paper that tries to pour academic scorn on our theory thus proving that the government is indeed trying to cover things up and suppress the truth.

We have evidence of such a coverup-in-progress with the paper by Summerhayes and Beeching which attempts to debunk the story of a Nazi secret base in Antarctica.

How can these two respectable adults deny that the Antarctic has been used by the National Socialists to create a secret base?   If it weren't true, why do they deny it so adamantly?  What else do they have to hide?   Perhaps the real secret lies in the Nazi base which may turn out to have been built atop the passage to the hollow earth.  Or perhaps instead of an immortal race of benevolent immortals with unlimited sources of energy, we find an ancient city built by an outcast race of elder gods not unrelated to Cthulhu?

Then all we would need to add to that is a secret society whose purpose is to keep the truth and thus the great power it confers to themselves, and we are pretty much all the way there.

I call on Summerhayes and Beeching to come clean about how they were coerced by this secret society and forced to lie about the esoteric knowledge that was revealed to them when they personally explored the abandoned Nazi base so many years ago, a visit that revealed to them an unspeakable horror that even now the world is not ready to hear.

The first three pages of the report

Pretty cool design for the expedition badge

You may read this scandalous suppression of the truth in its entirety here:

https://wikileaks.org/gifiles/attach/49/49783_.pdf

____________________________________________

The Nobel Peace Prize and Cultural Imperialism

10.11.2014

For the record, I am in favor of equal rights for women wherever they may be and I have no particular trouble with this prize, although I think it has a not so hidden agenda.  I do have a problem with people who think they have a right to fuck with other people's cultures because God is on their side, or because it is a "human rights issue".   Bunch of assholes if you ask me.

10.12.2004   19:30

On the one hand, I get a certain satisfaction out of blocking someone on Facebook. On the other hand it bothers me when people misunderstand my good will and are insulting and rude. If you do not agree, fine, you have a right to your opinion.   But to dismiss the argument because I am stupid or lame is just rude, and in this case we have a typical liberal who knows what is good for the whole fucking world and, cultural diversity notwithstanding, does not give a fuck what other people believe. He knows he is right, so very fucking right.

And indeed, I wish people would be more like us, or at least more like how I think we should be.  It would be better for the whole world if they did that.  But it is not going to be that way.

So now I am in the unenviable position of desiring vindication.  And vindication will only come if not only various sections of Pakistan or Islam respond in a negative way to this provocative prize, but possibly take disagreeable action.  I have no doubt that people, some people, are pissed as hell about it.  The question is whether they will do or say something such that even our inadequate press will notice and report it.   We will see.

The Nobel Peace prize has made it clear. Islam must give up all its politically incorrect beliefs and conform to the West's view of how the world ought to be.  Now if Islam will just give up its evil ways and conform to the politically correct views of the committee, how much better this world would be. 

How lucky the unwashed darkies are to have the superior west show them the way to improve their otherwise worthless lives and lead them to the path of righteousness.  I am sure that they appreciate all the effort expended on their behalf.

Cultural diversity as we all know is ok as long as everyone believes the way we do. Anyone who does not believe the way we do must be destroyed because our way is the right way and their way is the wrong way.

The very politically correct Nobel Prize Committee

And who is the Nobel Peace Prize committee?   Why it is five Swedes, three women, two men, all of them privileged white people.  Christians if they are anything.   Morally superior and perfectly capable to tell the world what is right and wrong.

I hope that the bad people have the grace to do as they are told.

http://nobelpeaceprize.org/en_gb/nomination_committee/members/

A special note to those who would comment, please try to avoid personal insults.  Yes, I know I am offending you by disagreeing with your right to tell people how to live their lives.  I wish that they (traditional conservative Islam as it is practiced in certain parts of the world) would educate their women better like the Egyptians always have (or at least some of them have).  But I think it is arrogant to tell people how to live their lives when there is so much more that can be done here at home to make the world a better place.

Economist Essay on the Future of Books

The Economist is still one of my favorite magazines. It is literate, it is functional, it has good analysis, and it is not as stupidly right wing as the Wall Street Journal has become. It is what we might call, intelligent middle of the road. The magazine is not about business alone, it is also perhaps about the world at large, but as seen through the eyes of an economist, hence its title.

But I am mentioning it today because it has an entertaining essay on the past and future of the “book”. And it has done so in a way that is cute: retaining the stains and other weirdnesses of the classic book.

Read the essay yourself at:

http://www.economist.com/news/essays/21623373-which-something-old-and-powerful-encountered-vault

Lies, Unemployment and the Bureau of Labor Statistics

        They lie. They lie and we have to be merciful with those who lie.

                 -- Col. Kurtz from Apocalypse Now (1979)

People demand different things from their government, but generally speaking our government is supposed to keep our nation out of a war when possible, to fight a war successfully when it is necessary, and to assure that their citizens are prosperous and free with valid elections, an honest judicial system, freedom of religion and a free press.

But when we have a failure of one of these criteria, such as prosperity, what should we expect from our government? Should we expect it to lie about the situation? Perhaps. Governments regularly lie about bad news. But should we want it to lie about the situation? Probably not. Probably when things are bad it is best for people to know about it so they can make reasonable plans and change the government if necessary. Obviously, governments do not in general want that, so they lie.

Its one thing to spin public opinion, its another thing to lie about something that is life and death. Its another thing to lie such that people do not have the information to make rational decisions, or to hide the failure of government, over and over again, to stay in power.  

I have a personal favorite in the lying sweepstakes, because it is so stupid, so self-serving, and so obviously indicates that our government is corrupt and without integrity, at least in this area. (1)

My favorite lie is the lie of the “unemployment rate” that is regularly announced by the US Government. For example, recently we heard that that unemployment rate has dropped to 6% and in some sense that is true. In other words, you can design a metric and then measure it and report it, but that does not mean anything about unemployment particularly. In this case, the metric is designed to not only misrepresent the situation but to do so in a way that is particularly deceptive.

Now those of us who have had the pleasure of studying aggregate economic indicators such as GNP understand that these things are far from perfect.   They are necessarily compromises that may have some value when properly interpreted.  Too much reliance on any one indicator is almost certainly a mistake.  However, when an indicator is deliberately designed to obscure what it is nominally supposed to reveal, then we cross a line from an imperfect tool to deliberate deception.  And that is what we find with the unemployment rate.

Because if you examine the details of what the unemployment rate measures, you discover that it only measures the unemployment of those who were recently employed. In other words, they had a job, they lost it, and then they got another job rather soon thereafter. But if for any reason they did not get a job again within 18 months, then they are no longer measured. They no longer exist on any statistic of unemployed as measured by our government so far as I know. (4)

These statistics come from the Bureau of Labor Statistics whose web site is www.bls.gov. If you have the time, you are invited to visit this web site and see if you can figure out what the unemployment rate means and how it is calculated. It will take a while but the information is there. And I have slightly oversimplified matters, but I am not wrong about the broad thrust. The unemployment rate deliberately understates unemployment, and in particular it deliberately understates those who are the most miserable and the most affected by unemployment.

And this is deliberate. Because you see, who is employed and who is not has very little to do with merit, but it has everything to do with fashion, with appearances, and with choosing a field that turns out to be less affected by globalization, or with foreign government subsidies, and so forth. And in the corporate view of things, which our government unanimously supports, only those who have been working recently are or should be eligible to be employed. If, for any reason, you have not been employed recently, then you fall out of fashion and become a much harder case to deal with.

And they don't want to deal with it. They don't want to know about it. They don't want you to know about it. They wish those people would go away and die, and in a sense they will. Because they are poor, and our government only responds to the rich, the unemployed are powerless to do anything about it. And so they become impoverished, a burden on their family, and their lives are ruined.

Because my career involves technology and media, particularly the use of computing in motion picture production, computer animation and visual effects,  I happen to know a number of people in the category of chronically unemployed. Of the dozen or so people that I know who are in this category and whose situation I have followed for a while, all but one of them are still unemployed three years later.  In some cases they have been unemployed for longer in spite of vigorous efforts to change that.   Several of them have had small projects during this period, which helps but does not solve the problem. A few of them have had jobs / projects that lasted a year and then finished.   All of my friends are interesting, educated, experienced and capable people with a good track record by the standards of the industry that they are a part of.  All but one of them failed to get a trust fund.  The problem does not lie with them, the problem lies with the structure of our economy. (2)

Of these people, three of them are essentially homeless, and one of them is literally homeless. (3) A fourth person is becoming homeless over the next few months.

And none of these people except the one who did indeed get a real job show up in the government's statistics on unemployment. And these people are trying and these people are miserable. But they are officially invisible.

The economy is not improving at least not for these dozen people I know. Furthermore, even if they were all employed tomorrow, or became a successful entrepreneur (which many of them have certainly been trying to do) nothing short of a making a whole lot of money will help. Why? Because they have accumulated debts, or lived on their retirement fund, and because they have not been able to add to their savings during these lost years of employment.

So even if they were employed starting tomorrow, they would in most cases still be fucked.

Which is why our government does not want to measure them, does not want to know about them, does not want to acknowledge they exist. To do so would be to admit that they, the government, has failed and lied all the while.

Now, the US Government, having lied about such an important issue, and having done so year after year, knowingly and deliberately, why should I or anyone ever believe them again?

___________________________________________________

Notes

1. In contrast, there may be areas where the government does have integrity.  For example, I have noticed and believe that when the US Government issues a terrorist warning, which is generally a vague statement that some sort of attack may happen in some area (e.g. W. Africa) that they only do that when they have genuine information that something might happen, but not enough to be able to stop it, or to be specific. In other words, I am accusing the US Government of integrity in this area. Other people believe that the government does this to manipulate public opinion or the results of elections, but I don't think so. Anything in the area of intelligence can be confusing to the general citizen outside that community, but I have to admit that trusting them in this area, but not trusting them in others may not be totally logical, or it may acknowledge an important fact: our government is diverse and one part is not like the other.  The FBI is not the NSA, and so forth.

2. In one case, the problem does lie with the individual who is so eccentric and difficult, if brilliant, that it is not to be believed. The fact that he ever held a job would be miraculous if you did not realize that the job he held was a complete anomaly and not at all a normal job in the least.

3. I call someone homeless if they are living on a friends couch or in a spare room but could not afford rent if this couch were not made available.  Then I call someone literally homeless if they do not have that couch, but are living on the street.

4. It may be that an indicator that may include such long term unemployed is the Social Security disability system.   If so, this is a very imperfect indicator, although I know that some people have found it a useful metric for this.

_____________________________________________________

Links

Bureau of Labor Statistics
www.bls.gov

Apocalypse Now (1979) on IMDB

http://www.imdb.com/title/tt0078788/

The Heavens Have Foretold Your Doom

At one time or another, many computer animation people have worked to create an illusion of the night sky from earth or of its cousin, a “star field”, which is an imaginary view of the stars from space. Whether this was for their own amusement, for visual effects purposes, or for scientific visualization, these innocents would approach the problem with the assumption that it was going to be easy. How hard could it be, its just a bunch of random white dots, after all. Imagine their surprise when they discovered that doing excellent starfields is far from trivial.

A classic traditional technique to create starfields is to create a cyc, or curved screen, painted black and with very small holes punched in it. Then behind this screen was a curved light source, usually florescent tubes. The camera would be at the center of the implied sphere of the screen and when the room was darkened and the backlight illuminated, you had a curved space of very bright, very small light sources which could be photographed with long exposures when the camera was moving. The result was excellent motion blurred, perfectly antialiased, very high contrast star fields. But ultimately there were certain moves that the motion control camera could not easily do, such as tumble end over end for example, so there was a need to synthetically generate these elements.

Another time honored technique which looked excellent was the painting on glass. Most of the times you saw stars in Close Encounters of the Third Kind (1977) you were seeing an optical composite of a live action element or motion control shot with a matte painting on glass.

Since everyone seems to have to go through the same learning curve, I am providing notes here for what some of the issues facing 3D technical directors as they produce their first starfield and I have written it as a letter to my younger self.

September 19, 1983

Oh, unwary traveler, so proud of your 3D knowledge, your knowledge of geometric modeling, or of animation whether scripted or procedural, and of global illumination; do you think to encompass the heavens with these pathetic tools? Fool, your doom is assured. There are more things in heaven and on earth than are encompassed in your philosophy, or so I have heard, and when you approach the field of scientific visualization you must unlearn what you have learned and embrace the esoteric wisdom. You must open your eyes in order to see the light.

What perils await the unwary, the arrogant, the unlettered?

The first peril is the vast expanse of space. There is the scale of mortal man, then the scale of the solar system, then the scale of one single galaxy, and then beyond. These differences in scales are way beyond what most software packages can handle, so using the 3D positions of everything in a naive fashion is unlikely to work.

And that renderer you are so proud of.  Does it do all its calculations of space in 64 bit floating point or even higher precision?   Most renderers, with a few notable exceptions,  do the majority of their work using single precision floating point which may be adequate for a giant robot or two, but falls apart in the vast distances of space.  

The second peril involves the issue of filtering of what is very untypical samples.  Most scenes render surfaces with various lighting applied.   But a great deal of what you wish to render are stars but what are stars? Stars are huge things, but they are (for all practical purposes) infinitely bright and infinitely small (on the screen). The amount of energy concentrated in a single pixel may be immense, but the pixel next to it may have very little or no energy at all. And what happens under those circumstances when you move the camera? Well, it aliases, of course, terribly. Furthermore, if one has modeled stars very far away and you are using point sampling of one form or another to simulate area sampling, then if you are not careful, some of your samples will miss and you will have aliasing again.

Part of the solution is to use a good filter and lots of samples and in the choice of filter lurks another threat since as we know a "good" filter, perhaps a 7x7 sinc for example, is likely to have negative lobes, and instead of throwing those values out, you should keep them until the end and even then you should not throw them away. What then to do with them is a mystery left as an exercise for the reader.  The best solution of course would be to have a display that could absorb light as well as emit it, but we wait in vain for the display manufacturers to come to our aid.

And what about those overly bright stars? Will you generate glows and other artifacts? After all we are not just trying to simulate realistic stars, we are often trying to simulate realistic stars as the audience has seen them, and expects to see them.

Although the sky is filled with stars, that is not the only thing that there is. There are also great fuzzy areas known as nebulae and sometimes other galaxies. It turns out that if there is any data for that, it is likely to be volume data. But even if there is no data and you create your own, volume rendering is the best way to render a nebula one might argue. Does your renderer of choice do volume rendering?

Review the following image of the earthling's galaxy.

Do you notice the great areas of darkness? That of course is the infamous "space dust", the so-called Interstellar Media or ISM which must surely exist to hide from us the center of our galaxy where no doubt an entity of great evil exists. Surely you do not think it a coincidence that the space dust would hide what is arguably the most spectacular sight in our little neighborhood? Since most star catalogs do not have the ISM modeled, you may wish to develop a model of ISM in your spare time. If not, the galaxy will not look right unless you simply leave out the stars that are in those areas (which may or may not be be in the catalog anyway as they are impossible to view from earth, at least in the visible bands).

Because you are rendering stars, no doubt you have studied scotopic vision.  It goes without saying that whenever the biped mammals have watched the stars they have, generally, been night adapted. And yet they see color sometimes, perhaps they see Angry Red Planet Mars or Betelgeuse and they perceive the color red.   How then are they seeing color? It may help the seeker of knowledge to realize that “scotopic” is named for the Skoptsy sect of religious devotees whose most notable doctrine is of male castration.  (see link below)

Of course I am sure when you move the camera you will motion blur everything. Oh yes, what do you plan to do with the speed of light issue? I am sure you will come up with something.

So, foolish mortal, you have been warned.

These are just the first of the issues you must address for a proper starfield.

Fools may go where wise people fear to tread.

Sincerely,
A Friend.

___________________________________________________________

Scotopic Vision

http://en.wikipedia.org/wiki/Scotopic_vision

The Skoptsy

http://en.wikipedia.org/wiki/Skoptsy

Close Encounters of the Third Kind (1977) on IMDB

http://www.imdb.com/title/tt0075860/

A New Star Catalog is Always a Cause for Celebration

A new star catalog is always a cause for celebration. Anyone who does more than dabble at astronomy visualization realizes that it is getting ahold of the data that is the sine qua non of serious work.

And yet getting the data is just the first of your many challenges. Very soon after downloading the data you will realize that one needs to have done some advanced study in astrophysics and/or have the services and advice of an astrophysicist or two in order to get anywhere. Without one or the other or both, the would-be visualizer will soon feel that he or she is lost in the forest on a moonless night.

But getting the data is the first necessary step, so it was with great pleasure that I read about the newly announced star survey with the catchy title of The second data release of the INT Photometric Hα Survey of the Northern Galactic Plane (IPHAS DR2)

And what is in this fabulous new survey?

The INT/WFC Photometric Hα Survey of the Northern Galactic Plane (IPHAS) is a 1800 deg2 imaging survey covering Galactic latitudes |b| < 5° and longitudes ℓ = 30°–215° in the r, i, and Hα filters using the Wide Field Camera (WFC) on the 2.5-m Isaac Newton Telescope (INT) in La Palma. We present the first quality-controlled and globally calibrated source catalogue derived from the survey, providing single-epoch photometry for 219 million unique sources across 92 per cent of the footprint. The observations were carried out between 2003 and 2012 at a median seeing of 1.1 arcsec (sampled at 0.33 arcsec pixel−1) and to a mean 5σ depth of 21.2 (r), 20.0 (i), and 20.3 (Hα) in the Vega magnitude system.

And so forth.

There are 219,000,000 unique sources in this survey. That is quite a bit more than the 10K star Yale Bright Star Catalog that we all grew up on. (1)

And yet there are certain similarities when you look a bit closer.   Certain subtle indications that suggest that the compilers of this fabulous expanse of data are adepts of the esoteric knowledge of astrophysics and that this data is intended for other members of this elite group.

In order to brace you for the tasks to come, should you decide to take on the challenge of visualizing this august data set, this post will present some first principles that are important for someone from the world of 3D as they step into the galaxy of creative decisions to be made and the arcane knowledge to master before the images latent in this esoteric compendium will be revealed.

I myself was first introduced to this knowledge during my brief but rewarding tenure on the Digital Galaxy Project portion of the rebuild of the Hayden Planetarium at the American Museum of Natural History in New York City.   Everyone on that project was a dedicated idealist with stars in their eyes.(2)

Because, you see, this dataset, like most astronomy datasets, is not intended to be for computer animation people. It is first and foremost to help scientists in the field, both those associated with the survey itself and those in the larger field of astrophysics, do their research. If people want to use it to “visualize the data” or make interesting pictures, that is all well and good, but that purpose is much further down the list of intended uses.

If we want to visualize the night sky what information would we ideally want? Well, we might want the 3D position of the stars, the brightness of the star, the color and presumably the type of star, as well as information about other objects in the sky that are not stars. This is a short list, there are actually many other things one might want to know for visualization purposes, but lets stop here. Ironically, a normal star survey or catalog does not generally have any of this information. At least not directly.

That is because what scientists are looking for in a star survey are the facts as they were observed. They may also be interested in your interpretations of the facts, but first and foremost they are interested in the data and how it was collected.

That means that instead of the 3D position of the “star” which we do not observe directly, except in a few exceptional cases, what the star catalog has is the position of the star as observed from earth on an imaginary 2D spherical coordinate system, a virtual sphere around the earth. In other words, the catalog does not say where the star is, but rather what direction it was observed to be from earth. The survey does not contain how bright the star was, or even if it was a star, but rather the spectra and intensity of the energy source as observed with the specific instrument. From that spectra, and from other information about the source of energy, one can deduce the type of star it probably is, but that is an interpretation of the data, not the data itself.

The star catalog or survey will not report the “color” as one normally thinks of such things, whether in RGB values, or CIE or other systems, but rather what was observed by the specific instrument or instruments used to collect the information. From the information in the catalog and the operating characteristics of the sensor one can derive spectra and from that a handy “color” in the visible bands if one so desires.

Nor is there any particular guarantee that the catalog is comprehensive, in some normal sense of the word. The authors of the catalog are not guaranteeing you that this survey contains all possible “stars” in the sky, or part of the sky. Instead what they are reporting here is the information about what they observed, and often some measure or estimate of how complete it is. The famous Messier catalog is not a complete list of anything, except the list of objects that Messier did not want to be bothered with anymore in his search for comets.

In a similar fashion, the data does not tell you how bright the star actually is, if it is a star, but tells you the intensity of the source as observed from earth. Again, an astrophysicist has various techniques to convert the apparent magnitude of the source to the absolute magnitude, if that is what you need.

The new star catalog has 219 million unique sources which may or may not be stars, and a variety of information about each, including some derived information (e.g. a probability that the specific source is indeed a star).

But it won't tell you anything about ISM (interstellar media), H2 regions and so forth.

When downloading the data, you can choose to download prepared datasets of the survey, or you can choose which data fields you want to review.  There are 99 different fields, and a dozen or so different formats you can get them in.  Just getting the data will probably take several days.

This post is the tip of the iceberg of what you need to know to do visualization from astronomy datasets but we will reserve these other topics for later posts.

The announcement can be found here

http://mnras.oxfordjournals.org/content/444/4/3230

The database can be found here

http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=IPHAS

______________________________________________

Notes

1. The Yale Bright Star Catalog is a famous dataset of the 10,000 or so brightest stars visible from earth. These are the stars that one could typically see from earth on a clear night with the naked eye from somewhere on the planet. It is a famous catalog in part because it was widely distributed even in days before the Internet or the Arpanet.

2. There were many, many people on this project which was part of a much larger project to rebuild the Hayden Planetarium and the entire north side of the American Museum of Natural History. Many people thought we were from Mars.  The cast of characters included such luminaries as Dennis Davison, Neil deGrasse Tyson, Carter Emmart, Frank Summers, Aram Friedman, Loretta Skeddle, Julio Morano, Benjy Benjamin, Ron Drimmel and many others.  And of course, your humble author, was also briefly a consultant on this project.