Python 3 Virtual Environments and Numpy

draft

Memo to myself.  These computers are no good.  I told you, but you wouldnt listen.  Now here you are, old, broken, poor and infested with Python version skew problems. Now listen to me for once and run to a virtual environment. For each project that uses python, create a virtual environment with 

cd project directory; python3 -m venv target-dir

Start the virtual environment with source target-dir/bin/activate

Install appropriate packages with pip3 install package-name

Next time maybe you will listen to me and not give me such a hard time.  I am only trying to help.

Potential Collapse of Civilizaton Seen as a Result of Weak Type Checking

In a world filled with the threat of war, with midterm elections that once again demonstrate the self-destructive credulity of the American people, with the collapse of the American economy due to the greed and stupidity of the American elites, is now the proper time to talk about the looming crisis of weak type checking in our programming languages?

Yes, now is the time. The need has never been greater to stop this foolish slide into moral decay and socialized health care.

The promise of weakly typed or untyped languages such as Javascript is that you can quickly and flexibly create new data structures, complex data structures, and not get bogged down by being forced to go through your entire system and make everything work together in a very pedantic, and literal way.  You can throw together arrays and lists and pass them as parameters and feel a certain pleasant lack of mental overhead in doing so.  

This can be very productive but it can also generate a false sense of correctness when in fact one has introduced a subtle incompatibility which the system is blandly ignoring for a while, only to fail in a non-trivial way when you are least expecting it.

In fact, it is shocking how well a system like Javascript is willing to accommodate undefined variables, functions, incompatible types and just keep moving along as if nothing was wrong.

But having seduced the programmer into a false sense of security, it then waits for the program to reach a certain size, or grow to more than a single programmer, and suddenly the author or authors of a program have to start tracking down a bug that comes from one side not enforcing or maintaining a data structure in the way to which it was intended, or partially implemented, or perhaps implemented but then changed due to incomplete knowledge.

The larger the system, the more people who contribute to a system, and the longer the software is in use and being evolved, the more likely this is to happen. And when it happens, one is left without the tools to find the problem other than reading the code carefully and providing ones' own type checking.

How could this lead to the end of civilization?   It can do so in two different ways.  The first is that by permitting this mental weakness, this accommodation to those who would advocate weak type safety, we are letting those who are more lazy enter into a position of responsibility in society. This is certain to lead inevitably to sloppy programming resulting in falling buildings, collapsed bridges, exploding nuclear power plants and God only knows what else.

But second, this nation is under relentless attack by inscrutable oriental criminal elements that are sponsored by their evil, slave owning, government.   Can you imagine their glee whenever they penetrate another freedom-loving web page or program in America that has been left defenseless by a weakly type-checked programming language?

We must stand firm against these efforts to leave America defenseless against these threats and rebuild American strength through strongly typed languages.

Thank you.

You Could Upload an Image from WEBGL or You Could Just Go Shoot Yourself

Lets say that you embrace the WebGL world and write some nifty graphics application in it.   Now lets suppose you want to save an image or two from your interactive application and doing a screengrab is not appropriate. Perhaps this is because you wish to save many images, such as an animation.   Maybe you are debugging a complicated problem and wish to save snapshots in various parts of a process.

Maybe you are iterating through a space and need to make a record of each step.  I do this all the time in animation production to do what is called a "wedge", which is a way of making choices among many parameters.  

But this is WEBGL and the bold new Internet paradigm, so things are not so easy.

In fact, its a really annoying process to save an image under program control and the reason for this has less to do with WebGL than it does to do with Internet architecture and the awful state of documentation in our bold new Internet based reality.

I present here one solution to the "save an image" problem.  Here are choices that had to be made to get to this solution:

1. In order to save an image, you must upload it to a server.
2. In order to upload anything you must use "Ajax", which really means XMHHttpRequest.
3. The server side is written in node.js
4. We are going to use POST.
5. We are going to use base64.
6. We are not going to use FORMS or BLOBS.
7. Our server is going to do nothing but receive an image and save it out.
8. You may save as many images as you like, the filename will be incremented by 1 each time.
9. All images are saved in PNG format.
10. We are going to use getDataURL() directly from the canvas and not use gl.readPixels at all.
11. Therefore we do not have to set any special flags when we acquire the 3D canvas.
12. If a file already exists of the same name, it will be written over without comment.
13. The images are put in the directory that you run the node server out of.

If you are new to this, these choices made above eliminate a billion or so other possible solutions and therefore makes the problem solvable (instead of iterating through an infinite search space getting half baked quasi solutions on the internet).

When it is all said and done, the result is about 1/2 page of code on the browser/client side and about 1 page or a little more on the server side.  Everything is written in Javascript, with the server side making use of node.js.

Here is the code that works for me.  If you try it and it does not work for you, please let me know.   

Uploading An Image (Client/Browser Side)

When the 3D canvas has an image you wish to save, do the following: 

// generate "dataURL" for the image on the screen

// "canvas" is what is returned from document.getElementById on your webgl canvas

   var newimg_dataurl = canvas.toDataURL("image/png"); 

// NB  The server is going to be looking on socket 8080 and a path of  /imagehandler.  

// But these are arbitrary and can be whatever you like 

   s_postimage(newimg_dataurl, "image/png", "http://localhost:8080/imagehandler"); 

// the function that actually posts the image 

var s_postimage = function(dataurl, imagetype, dest_url) {

    var xr = XMLHttpRequest(); 

    xr.addEventListener("error", xfer_error, false); 

    xr.addEventListener("load", xfer_complete, false); 

    xr.open("POST", dest_url); 

    xr.send(dataurl);

    return;

}; 

var xfer_complete = function(e) {

    // document.write("<br>xfer complete");

}; 

var xfer_error = function(e) {

    if (e) throw e; 

}; 

Uploading an Image (Server Side)

This is run with the command "node imageup.js" where the file imageup.js contains:

var http = require('http');

var url = require('url'); 

var querystring = require('querystring');

var util = require('util'); 

var fs = require('fs'); 

var img_seqno = 0; 

http.createServer(function (req, res) {

    switch(req.url) {

     // This is where you would put in handlers for situations/requests other than 

      // the request to upload an image

    case '/imagehandler':

if (req.method == 'POST') {

           console.log("[200] " + req.method + " to " + req.url);

           var fullBody = "'';

           var fname = "upimage_" + img_seqno.toString() + ".png"; 

           console.log(fname); 

           req.on('data', function(chunk) {

          // append the current chunk of data to the fullBody variable

              fullBody += chunk;

             console.log("data received"); 

});

        req.on('end', function() {

           // request ended -> do something with the data

           res.writeHead(200, "OK", {'Content-Type': 'text/html'});

           console.log("end received"); 

           var base64Data = fullBody.replace(/^data:image\/png;base64,/,"");

           fs.writeFile(fname, base64Data, 'base64', function(err) { 

              if (err) throw err; 

             console.log("image saved " + fname); 

             img_seqno ++; 

}); 

        res.end();

});

} else if (req.method == 'OPTIONS') {

            var headers = {};

            headers["Access-Control-Allow-Origin"] = "*"; 

            headers["Access-Control-Allow-Methods"] = "PUT, POST, GET, DELETE, OPTIONS"; 

             headers["Access-Control-Allow-Credentials"] = false;

             headers["Access-Control-Max-Age"] = '86400'; // 24 hours

             headers["Access-Control-Allow-Headers"] = 

                 "X-Requested-With, X-HTTP-Method-Override, Content-Type, Accept";

            res.writeHead(200, headers); 

            res.end(); 

            console.log("OPTIONS received"); 

        } else {

            console.log("[405] " + req.method + " to " + req.url);

            res.writeHead(405, "Method not supported", {'Content-Type': 'text/html'});

            res.end('<html><body>Method not supported</body></html>');

}

break;

    default:

        res.writeHead(404, "Not found", {'Content-Type': 'text/html'});

      res.end('<html><body>Not found</body></html>');

console.log("[404] " + req.method + " to " + req.url);

    };

}).listen(8080, "127.0.0.1"); // listen on tcp port 8080 on the localhost

// end of server code

More Balanced Commentary on WebGL

No matter how frustrating the process, after a while one learns what can be done with a system and how to get around whatever it is you need to get around.   In this case the surprise was that I simply did not realize to what extent programming WebGL meant actually also working with both Html and the DOM, even though in retrospect it is obvious, and even more than that, this is what you would want.

You want it to be involved with HTML and the DOM or how could it all be integrated into the browser otherwise.

So its all good.  

There are some mysteries in the direction that OpenGL ES 2.0 has taken.   These mysteries can no doubt be explained by the designers, but they haven't done so, not that I have been able to find.  The specific questions involve the extreme position they seem to have taken to remove many things that people who program graphics use.   Its not so hard to implement your own versions of these things, e.g. a perspective matrix, a uniform variable to put it in, etc.   But it does make one wonder why they took it out to begin with?   Maybe they are just trying to be minimal?   That could be good.  But it is a little puzzling when you know the earlier versions of OpenGL and just leap into this and can not find things you expect.

Same question about perspective matrices also applies to lights.

And they seem to have some major browser architecture issue which I will call "the problem of the lost context".   Its annoying but can be managed.

On the positive side, by being so integrated with Html and DOM it is trivial to do multiple window layout and such things as scrolling text windows that might otherwise be difficult.

So far it seems to work and browser incompatibilities between Firefox and Chrome have been minimal.

The Failure of Moral Progress: The Tragedy of Javascript

From hope to despair, then to hope, then to despair again, is that the fate of all civilizations? To give in to sloth and decay, their monuments to reason covered with the slime of intellectual and moral weakness? To sink again and again into corruption, incompetence, venality and death ? Is hope for progress mere pablum for the weak minded to keep them enthusiastically at their tasks until it is too late and their fate is sealed?

Something has been revealed to me recently that would make me think so.

A computer language is many things but one of them is a (usually formal) specification of a grammar and a syntax that is useful to the bipeds in expressing their ways of doing things, what are sometimes called algorithms, named for the jazz trio of Johnny von Neumann, Alan Turing and Alonzo Church whose band, The Algorithms, dominated avant garde jazz in the 1930s up into the 1950s and whose influence is still heard today. Although writing computer algorithms is a very personal and idiosyncratic form of expression, the notation that the individual artists (and groups of artists) use to express themselves will subtly affect the elegance of the algorithm and can by its nature guide and channel what can be expressed. They may all be Turing equivalent at some theoretical level but they "feel" very different.

There is no one such formal language, there are many, and there will be many in the future. Like music and music notation, they will evolve and some will be appreciated by an elite, and some will be used by the masses. Some, like SNOBOL are esteemed but not in current usage. Others like C++ (pronounced "C Double Cross") are as common as flies on shit and just as attractive.

As in all things there is the matter of taste and the issues of elite style vs common style. The avant garde must by its very definition be avant, changing and moving forward.

Even so, we can look on in horror or at least puzzlement when something that is fundamentally flawed, something that we know is just not going to be good, becomes established and then through the vicissitudes of the uncaring fates explodes onto a hundred million computer screens to become encrusted into just as many computer programs and taught to our children and then to their children in perpetuity.

I have just looked more closely at HTML 5 which is already everywhere and soon will be truly everywhere. One day there will be an HTML 6 no doubt but until then it is HTML 5 that will be used to mark up what our civilization has to say about itself. HTML 5 is a synonym for Javascript, as Javascript is integrated into the very essence of HTML 5. There can not be one without the other. Where you find HTML 5 you will find Javascript.

The more I learn Javascript the more I realize that Satan and the Illuminati, another band from the 1930s, must be chortling with glee at the little joke they have played on our world. For Javascript is a pastiche which pulls a little from column A and column B and column C and Java and Scheme and C and blows smoke in our face. It is a tale told by a billionaire, Mark Andresson, who was in a hurry at the time and would we have done any better if we were in his shoes?     I would hope that we would, but it is very hard to know until we are tested, and we probably never will have that opportunity.   It is what it is, however.

Javascript is not the best we, the computer community, can do in a perfect world.  But it is not a perfect world, and at least Javascript is not the worst that there is out there.    At this point, it is just a fact of life.

___________________________________

HTML 5 Working Group
http://www.w3.org/html/wg/

HTML 5 on Wikipedia
http://en.wikipedia.org/wiki/HTML5

Lots of Data

One difference between now and then, between the world before the WWW and now, is access to data. It used to be that there were not that many data sets that were online and available. Just collecting the data, or inputing the data was a task of monumental scale. But today, we have detailed maps of the United States down to the most amazing level of detail readily available. A variety of people must have worked on that project for years and years, and there must be hundreds, if not thousands, of people working to maintain accuracy of that data as it changes with time. But we do not see that and, somehow, we do not seem to have to pay for it. Someone does, but not us.

It used to be that certain groups would maintain their datasets and you could get a copy by sending them a tape. Astronomy catalogs are a famous example. We all used the YBSC (Yale Bright Star Catalog) because that is what was readily available. Now there are many catalogs available, easily, on the Internet. Of course, that does not mean it is all that easy to interpret that data, but that is a different (if related) problem.

By accident, while my life is collapsing around my head, and the future looks as dark as it ever has, I was wasting time as I usually do and came across a remarkably useful collection of datasets. It seems to be associated with the Introduction to Computer Science course at Princeton University.

You can find that dataset in its original form here. 

War and Peace, what we normally call "the Bible", the works of Dickens, the Official Scrabble Dictionary, a few human chromosomes, the ranked list of last names in America, all the locations of Wendy's burgers in the world, and so forth. The Communist Manifesto. Its all here.

The table of contents, in image form, is:

Now you will never need data again.

I plan to download all these and make a giant tar available on the internet of all of them.  Why? Because I believe that everything that exists on the WWW will go away, one day, whether we know it or not.