Notes on Installing Windows 10 and Google Tensorflow (CPU only)

draft

This is the third time I have installed Windows 10.  These are my notes from the experience.

1. Based on previous disaster, all other disks were unpowered when installing this OS.  The only disk it could get to was the one that it was to be installed on.

2. Based on previous disaster, this disk is connected by SATA because the poor little piece of shit, Windows, cant install on  USB (a marketing decision we are sure).

3. You need a Microsoft account to do this.

4. You sign an agreement which says in so many words that you are stupid for running MS software and if anything goes wrong you should go fuck yourself.

5. It reboots itself without telling you to remove the USB boot disk.

6. You must give it a device PIN.  

7. You do have the option of turning off Windows Cloud, Cortina and weird non-privacy options.

8. It does not let you change the resolution on your screen.

9. When you install Firefox, it complains and tries to make you use its own stupid browser because of course it does.  This is Microsoft.

10. Even though you turned Cortina and MS Cloud off it is still running, because of course it is.

11. When you download the Nvidia driver you have no choice but to load the "GEFORCE Experience" and wait 20 minutes while it downloads. The driver itself is at most a 1 minute download.   Hey dont worry, my time doesnt matter.

12. One of the things I find fascinating is how slow it is.  It is running on 8/16 fairly fast processors and essentially infinite memory, on an SSD drive at SATA 3 speeds and it is still slow.

13. Installing the NVIDIA driver was trivial, but ... here I am waiting for it to reboot.  Will it reboot?  Will it ever leave the SETUP screen?  Oh Oh.  Second reboot and it comes up just fine on the Nvidia card.

14. Must sign in with PIN.  Weird.  No way to turn it off that is obvious.  Whatever they think they are doing with their control panel settings, it isnt helping.  I cant find anything.

15. Now to install VLC for Windows (my preferred video player) and then we get to the good stuff.  Tensorflow and Unity.  OK VLC worked but forgot to turn off metadata download.  Later.

16. Tensorflow ... two versions... CPU only and w/GPU.  It says that beginners should only do CPU.  So I install pip, virtual env, follow instructions for virtual env (one mistake in doc), run virtual env, load tensorflow via pip, run it and it fails!  But wait, it fails because I skipped a step, the "MS Visual Development blahdy blahdy blahdy".

17. So I sign up for visual developer, which means a c/c++ compiler I think, and it also has a python development system and it tries to download 2 GBs.  So I pause it, cut off everything but what I need the core and python, and that is a mere 380 MBs.  Fine.  Installed.

18. Reboot.  Need to reboot twice to get Windows to run. Second time this has happened weird.

19.  But now when I run tensorflow (cpu only) in python, in virtual environment, it works!  Yay!

20. Installed matplotlib.

21. OK we are almost where we were a month ago.

22. Science Marches On!

Tensorflow on Hold After Weeks of Wasted Effort

draft

Time that could have been spent learning machine learning has instead been spent trying to get Tensorflow to work on Centos 7.5.  Its been a neverending nightmare, a labyrinth of conflicting requirements, installation programs, neurotic programs that require specific versions, complications of using a GPU and so forth.  I feel like I am living the Reagan "Star Wars" nightmare of software that is one bridge too far.

Let there be no doubt I made this worse for myself in a variety of ways.  This computer has to serve a lot of purposes, not just one.  Google Tensorflow makes a precompiled version for Ubuntu, which I despise, and makes the source available for other Linux's including Centos (Centos == Redhat for those of you outside that world).

Without doubt, the biggest problem here involves Python and its wacky Pip installer.  Look, I know that if you are immersed in the Python culture that this all works for you kids.  But I am not immerse, I want to use Keras and Tensorflow, Python is just along for the ride.

I had it working for normal CPU work and it was painful but not impossible.  When I went to use it with a GPU, though, it all fell apart and now I can not get back.

The good news is that I do have CUDA working, which is a non trivial accomplishment.  Of course, X is broken.

This nightmare has to end.

Centos Linux 6.6 Good, 7.0 Bad

When a new major release of an operating system comes out, I generally wait six months or a year, and then try installing it on a non-critical, non-production machine that I keep for just this purpose. That way I can keep doing serious work on my main systems while I work out the issues with the latest release.

In the past this has worked out well for me.

I have been very pleased with Linux as a workstation environment as well as a production server environment. Although not real Unix, it has certainly been very reliable and surprisingly scalable, which is remarkable I think.

For those of you who do not know your Linux distributions, Centos is basically the same as the major Redhat releases, but without the Redhat logo and without the formal support (for which of course you are expected to pay).

It has lots of people working on it both at Redhat and out in the world and although it has its quirks, as all Linux distributions do, I have had really very little to complain about beyond the usual issues that one faces when there are too many solutions to a problem and it is not clear which one to use.

For the first time, I have tried a new release of Centos and backed off to the previous release. Several things happened to cause me to do this.

The major annoyances all came down to the situation that the desktop part of Linux was not getting the attention that those of us who use Linux as a development environment would desire. In particular, the X window system does not automatically come up any more, and you have to jump through hoops to try and make it automatically start. Also, a variety of features that I have been pleased with in Gnome (one of the two desktop environments) suddenly went missing. Third, there were serious performance issues after a period of use that I could not explain, and did not seem to have anything to do with the usual things one looks at in tuning a system. It made the system both cranky and unreliable.

But the final disaster that just was unacceptable, is that this release of Linux insists on installing system patches and updates whether or not you want it to, destroying the old version, and imposing the new version on you. The problem is that it did so, but did not deliver a working version of Linux. The system had been automatically trashed and I then had to decide how I wanted to recover when there was no easy fix. Some of this brought on because of brain damage of the Linux community involving graphics drivers. I use the Nvidia driver exclusively and that may have complicated things.

But it is an ironclad rule here at my place of work, that updates are not installed until I want them to be installed just to avoid this kind of problem. This is not an isolated incident. I have in the past had very bad experiences where kernel updates were made and the OS stopped being usable. But in this case it is not obvious how to turn off the automatic updates and I dont want to fuck with it.

So for my uses Centos 6.6 is infinitely preferable to Centos 7. It comes with a working window system, a working desktop, is more reliable, and doesnt self destruct whenever it wishes to.

I dont know what this means for the future, and that does worry me.

Google Chrome Now Works with Centos Linux

I have used Centos Linux for all my professional software development in the last five years or so and I have been very happy with it. For those who are sensitive to the different camps of Linux self-Balkanization, this is in the same lineage as Redhat and Fedora.  It is supposedly the most used variant of Linux for commercial and enterprise-related activities.

But one problem with it is that it was very difficult to get Google Chrome to run under Centos even though it seemed to work fine with Ubuntu.

Anyway, this has been fixed, and it is necessary to run a script which has been created by Richard Lloyd, and you may read about the situation and the fix here.

This is known to work under Centos 6.5 which is the version I am running.

Thank you, Mr. Lloyd.

http://chrome.richardlloyd.org.uk/

The Infinite Mystery of NVIDIA Linux Device Drivers 2/2

In our previous post we outlined what you have to do to install the NVIDIA graphics device driver under Centos / Redhat Linux 6.5 and similar systems.

In this post we go over each item in boring detail.

I am pretty sure there is a mistake in here somewhere (builds confidence, doesn't it?) because I distinctly recall doing an /sbin/telinit 3 and 5 at some point in the process, but obviously I have edited this out of my longer term memory.  But have no fears, if this doesn't work exactly I am sure you will get the general idea and improvise as necessary.  You have my greatest faith in your abilities.

Remember, before you begin you are expected to sacrifice a goat at least and say the proper prayers spreading the prayer cloth and so forth.

1. Are you already running the NVIDIA driver? How can you tell for sure?

The way I do this is by running "nvidia-settings" from the shell. If a window comes up, everything is installed and working. If it can't find it or it doesn't run, you do not have the driver installed and should proceed.

    nvidia-settings

2. Install the development tools.

Become su and run the command

    yum groupinstall "Development Tools" --skip-broken

Notice the quotes and capitalization. Say yes when it asks you if its ok. Then go get coffee and come back in 15 minutes.

3. Figure out the model of your graphics card.

As su, use the command

    /sbin/lspci -nn | grep VGA

The reported line will have the model of the card. It will probably begin with the word "Geforce" or "Quadro".

4. Figure out if your Linux is 32 or 64 bit

Run the command

    uname -m

and it will tell you.

5. Download the driver from Nvidia.

With your browser of choice, navigate to www.nvidia.com and click on "Drivers". You should find a menu which will ask you which OS you are running (e.g. Mac, Windows, Linux 32, Linux 64), which series of graphics card (e.g. 9 series, 8 series, 200 series, 600 series, etc) and the specific model.  Fill these in and hit search and it will find your driver. A 9xxx card is the 9 series, but a 4xx card is a 400 series.  If you do not find the model of your card then you gave it the wrong series and you should try again.

Click yes or agree on the license agreement and save the driver some place you can find it again. Get another cup of coffee and go play with your dog or something. Come back in a few minutes.

6. Put that driver someplace easy to get to, like /tmp and make it executable.

I used to skip this step but it was too much work. Copy the file you just downloaded to someplace easy to type, like /tmp. Then make it executable with the command

    cp NV* /tmp

    chmod 755 NV*

Ok, now we get to the weirder stuff.

7. Change /etc/inittab to bring the system up at level 3

    su

    cd /etc

    cp inittab inittab.orig

    vi inittab

and change the "5" in the last line to "3".

8. Blacklist Nouveau, Part 1

    su

    cd /etc/modprobe.d

    cp blacklist.conf blacklist.conf.orig

    vi blacklist.conf

and add three lines to the bottom of the file which say

    # get rid of nouveau driver

    blacklist nouveau

    options nouveau modeset=0

and save the file.

9. Blacklist Nouveau, Part 2

    su

    cd /boot/grub

    cp grub.conf grub.conf.orig

    vi grub.conf

and add the word "rdblacklist=nouveau" to the end of the lines that specify the kernel to boot. These are the longest lines in the file and begin with the phrase "kernel /vmlinux-blah-blah-blah".

Dont fuck this up or you wont be able to reboot the system without some major help.

10. Reboot the system

11. Log in as root

12. Verify the location of the system kernel libraries

    cd /usr/src/kernels

    ls

You should see some horrible name that starts with 2.6. You either want to remember that name, or if you are like me, you want to save it as a shell variable. I use csh or tcsh, if you don't then you will have to figure out how your shell does this.

    cd 2.6<esc>

    set kernelpath=`pwd`

Notice the funny direction of the quotes on that set command.

13. Install the Nvidia driver

You should still be root. Change directory to /tmp or where you put the nvidia driver and say

    cd /tmp

    ./NV<esc> --kernel-source-path=$kernelpath

You will need to use the mouse, agree to the license agreement, and watch with your breath held while it jumps through hoops. When it asks you if it can create a new X11.conf say yes. When it asks you if it can install 32 bit compatibility say yes.

If it completes without complaining much we are almost done.

14. Restore the /etc/inittab runlevel to 5

Still as root say

    cd /etc

    cp inittab.orig inittab

It will ask you if its ok to overwrite a file, say yes.

15. Reboot the system

Hopefully the system will come up and the window system will come up, and you will log in.

16. As su, run "nvidia-xconfig"

    su

    nvidia-xconfig

17. Run "nvidia-settings" and save your work.

Say

    nvidia-settings

and goof around with the different options and screens and set things to your liking. Or just say ok and quit.

That is it.

You are done.

The Infinite Mystery of NVIDIA Linux Device Drivers 1/2

We are now going to do a series of boring technical notes that nevertheless fulfill a more noble purpose.

Those of us who are interested in doing or fostering independent production with computers and computer animation have a lot to be grateful for with Linux, particularly with Centos/Redhat Linux. It is stable, scalable, various applications work with it and it is remarkably Unix-like for something that isn't Unix.

And it has so far avoided the insane Linux-of-the-day bullshit of Ubuntu, praise Allah.

But we do not live in a perfect world and a well-tempered Linux is going to be running the NVIDIA device driver with a solid, respectable NVIDIA graphics card and we are going to discuss why and then dive into the crazy shit you have to do to make it all work.

The only manufacturer out there that really supports Linux is NVIDIA, in spite of what you may have heard of Linus fingering NVIDIA. His frustration is real, but it is bullshit. For a variety of reasons NVIDIA is not able to put their device drivers in the open source because it contains proprietary information they just do not want to give their competitors. So that means that certain purists can not include it with the standard distributions. That means you have to install it yourself.

With Centos / Redhat, one also has the option of using the so-called "nouveau" driver which supposedly supports NVIDIA. I have crashed my system so many times using the nouveau driver that I do not recommend you use it for serious work. Since I also do things like CUDA/GPU development, it isnt a serious option anyway.

So we are going to install the NVIDIA driver, and it is a little wacky to do so. There are three fundamental reasons why installing the driver is so convoluted and they are (a) you can not install a graphics driver if you have graphics running, (b) in particular you have to get rid of the nouveau driver and its non-trivial to do so, and finally (c) you have to be able to "compile" the NVIDIA driver against your current Linux and that means having the development environment loaded.

There is one more reason too that this note was written. There is so much craziness on the Internet about what to do with graphics drivers that I am adding my own opinion to the mix, along with clear instructions.

In order to follow these steps, you need to know how to do basic Unix/Linux things like ls, cd, chmod and so forth, have the su/root password, be able to use a text editor like vi that can work without X running, have access to the Internet, be able to use a web browser to download files, and be able to follow instructions.

In this post, I am just going to outline what you will have to do to install the driver. In the next post, I will fill in the details of how to actually do these things.

So hold on everybody, here we go!

1. Are you already running the NVIDIA driver? How can you tell for sure?

2. Install the development tools.

3. Figure out the model of your graphics card.

4. Figure out if your Linux is 32 or 64 bit

5. Download the driver from NVIDIA.

6. Put that driver someplace easy to get to, like /tmp and make it executable.

Now we get into the more tricky stuff.

7. Change /etc/inittab to bring the system up at level 3

8. Blacklist Nouveau, Part 1

9. Blacklist Nouveau, Part 2

10. Reboot

11. Log in as root

12. Verify the location of the system kernel libraries

13. Install the Nvidia driver (say yes to compatibility mode if you are asked)

14. Restore the /etc/inittab runlevel to 5

15. Reboot the system

We are almost done, hang in there!

16. As su, run "nvidia-xconfig"

17. Run "nvidia-settings" and save your work.

My goodness, that was annoying wasn't it?

The next post will have the mere details of how you actually do these things. The only thing really, really arcane is the blacklist of Nouveau.

Ubuntu Linux Now Controlled / Owned by the People's Republic of China

For those of you who care, Ubuntu Linux is now for all practical purposes owned by the People's Republic of China. I doubt very much if you care, but since we are in a war with the PRC (an undeclared one), the PRC will be putting a variety of trap doors and other compromises into the system. Now the fact is that it is very unlikely that anyone who reads my blog is going to be doing anything serious for national security, which will no longer be able to use Ubuntu, so it doesn't really matter.

Of course if you do proprietary technical work, you should think twice about using Ubuntu Linux because pretty much any Chinese hacker can get access to it.   Of course that might be true using any OS given all the energy the Chinese put into stealing intellectual property.

Just thought you should know.

Here is an article that goes over the public elements of the deal.

http://www.theregister.co.uk/2013/03/22/china_makes_linux_os_with_canonical_help/

Linux Enhanced Precision Firearm Experience: Sniper Rifle Goes Open Source

Now open source operating systems, lasers, image processing and firearms are combined into a single consumer product: Tracking Point has a new rifle that integrates various techniques to increase the accuracy of your average sniper / long range rifle user.

What I want to know is whether this will run Ubuntu and whether one will always have to constantly upgrade the OS or "be left behind". I would hate to update the OS on this thing and have a problem, you know what I mean? Ooops, sorry, didnt mean to shoot you!   Ha ha.  Sorry about your dog.

Whats the upgrade system on this distribution?  Yum ?  Apt ?   How do I know if the file system is reliable, is it journaling or using ZFS ?   Has GLUT been ported to this platform?  What kind of boot loader does it use?   Are we talking LILO or GRUB here?   How can I get to the shell?  I wouldnt want to have a sniper rifle for which I could not get to the shell.

Does Open Source Software have a role to play in homeland security and national defense?  I would hope that surveillance cameras, unmanned drones and personal nuclear weapons would all run Linux and presumably Apache.   That should teach a lesson to those liberal do-gooders who thought that Open Source was going to change the world.

Read about it at Ammoland

http://www.ammoland.com/2013/01/trackingpoint-launches-worlds-first-precision-guided-firearms/#axzz2PJzFgmce

Here is the video which describes what they are doing:

http://www.youtube.com/watch?v=LvbyAcYjzlc

And of course, each rifle comes with a WIFI hotspot to foster communication between itself and other weapons peripherals.   

Editorial

http://techcrunch.com/2013/04/02/a-precision-guided-firearm-powered-by-linux/

The Manufacturer

http://tracking-point.com/