Data Processing

I have been watching many videos on early-era computing and they have led me to give a lot of thought to the fundamental concepts in computing we now take for granted.

For example, sometimes we say “data processing” when we speak about our field, but the origin of that term is often long forgotten. The first computers were not used for watching videos or browsing websites; Instead, they were primarily used for solving large mathematical functions that would have previously taken teams of humans weeks, if not months, to calculate. These computing systems were single-task machines and had about the same general functionality as a modern Central Processing Unit (CPU). “Memory,” as we presently think of it, was in short supply and data was stored on punch cards. One of the first major uses of a computing system was to help tabulate the United States Census. It sped the entire process up tremendously.

Put simply, computers at that time were used to process data. That is all they did. That is all they were capable of doing. It was not for many years and decades that we were we able to build on this core concept to arrive at the computing systems we have today. All of which is built on processing data, manipulating it, and storing it again. 

Upgrading Ubuntu from 17.04 to 17.10

In June of 2017 I upgraded my Ubuntu 16.10 (Yakkety Yak) LTS install to Ubuntu 17.04 (Zesty Zapus). In October of 2017 Ubuntu released 17.10. (Artful Aardvark). I had not needed to upgrade my Ubuntu 17.04 installation for a while, but I felt it was best to keep it up-to-date. I was also experiencing issues with it from day-to-day. I decided an upgrade might work to fix some of the issues I was having.

The upgrade process is still fairly simple, but when performing it over SSH it can still be complicated. I ran into a small snag following my old procedure, so I felt it would be good to adapt it to a more generic upgrade procedure. As before, the process is not set-it-and-forget-it automated–it should be attended. SSH should already be configured and in-use, if this is your intended method, as well. This process assumes you normally access the system via SSH, but for those who do not know how, here is how to setup SSH.

Special Concerns

If the intended system on which the upgrade will be performed is a production system or contains valuable data, please consider performing a backup. There are several methods by which to accomplish this task listed here.

1. Install Screens

First off, make sure your repositories are up-to-date in your current distribution by running:

sudo apt-get update

Now we can install screen:

sudo apt-get install screen

Enter screen for the first time by typing:

screen

and pressing Space key for next page or Enter key to end, in order to accept the license. You can learn more on how to use screen here. The commands to run screen for this process will be included below.

2. Check for Release and Set to Normal Release Distribution

Check if there is even an update available to you by typing the following command:

do-release-upgrade -c

If a newer version of Ubuntu is available, it will be shown in the returned results. If this command does not work, you will need to install the Update Manager by running the following command:

sudo apt-get install update-manager-core

After the package installs, check for the upgrade again:

do-release-upgrade -c

If there is no version available, we need to make sure we are setup to upgrade to the latest normal release, which requires a small edit to a system file. To make this edit, type the following:

sudo nano /etc/update-manager/release-upgrades

Find the line of the document called prompt and make sure it says prompt=normal. If it says prompt=lts please change it.

If changes were made, press ctrl+o to save changes and then ctrl+x to exit. If no changes, just press ctrl+x.

Check for a new version again:

do-release-upgrade -c

If there is not one, your distribution may be too old and you may have to consider upgrading manually to a newer version.

Beware of PPAs

Run the following command to check your repositories listed on the system:

grep -r --include '*.list' '^deb ' /etc/apt/sources.list /etc/apt/sources.list.d/

-or-

grep -r --include '*.list' '^deb ' /etc/apt/ | sed -re 's/^\/etc\/apt\/sources\.list((\.d\/)?|(:)?)//' -e 's/(.*\.list):/\[\1\] /' -e 's/deb http:\/\/ppa.launchpad.net\/(.*?)\/ubuntu .*/ppa:\1/'

If you have PPAs on the system, they may interfere with the upgrade. Consider removing them and returning any packages to default from the support repositories with the following package:

sudo apt-get install ppa-purge

After it installs, remove the PPAs manually with the following command:

sudo ppa-purge ppa-name

Replace ppa-name with name of PPA Repository.

3. Start a New Screen and Upgrade

At this point we need to start a new screen for the upgrade process, because the upgrade process will kill the current SSH session. To do so, type the following commands:

screen -S upgrade

This will drop you into what seems like a new terminal session. In this screen type:

sudo do-release-upgrade

IMPORTANT: When going through the upgrade process you will be given a new port on which SSH will function during the upgrade. Document this number–in my case it was 1022. CANCEL the upgrade once you find the new port.

We need to edit the firewall in order to allow access to the host on the new port by running the following command:

sudo iptables -I INPUT -p tcp --dport PORT# -j ACCEPT

Replace PORT# with the new port number presented in the initial part of the canceled upgrade process.

To resume the upgrade, run this command:

sudo do-release-upgrade

Go through the process until you lose SSH access to the session and then keep following this guide.

4. Re-Establish SSH Access

Once you lose access to the default SSH port during the upgrade, you will have to use the new port number the upgrade process opened to attend the upgrade process. Complete the following steps:

ssh -p PORT# USERNAME@HOSTorIP

Once the new SSH connection is established on the new specified port, complete the following commands to attend the upgrade process:

screen -d
screen -r upgrade

5. Attended Upgrade

There will be multiple prompts during this upgrade process, so it is recommended you sit with it and periodically check it. The upgrade took me roughly 30 minutes in total on a 50 Mbps connection. The download will be roughly 1.4 GB in size, so the connection speed can make this process vary in time drastically.

I wish you the best of luck with the upgrade! Let me know in the comments what your experience was like.

Checking Disks in Linux

To provide a little background: a few months back I accidentally washed a 32 GB flash drive. I waited a few weeks for it to completely dry out and then did not use it for almost four months. I formatted it recently in Windows and it did not seem to exhibit any issues, but I wanted to know with more assurance that it was reliable. Continue reading Checking Disks in Linux

How to Upgrade Ubuntu Distribution Versions in 5 Simple Steps

A few days ago I was attempting to install the OpenSCAP security compliance evaluation tool on a system I had running Ubuntu 16.10 (Yakkety Yak) LTS. The base package of OpenSCAP was in my distribution’s repositories and I was able to easily install it. But, it turns out the SCAP Workbench package, which I also really wanted, was only available in Ubuntu 17.04 (Zesty Zapus). My choices were to attempt to build from source with directions for Fedora or upgrade my current Ubuntu distribution to 17.04. Continue reading How to Upgrade Ubuntu Distribution Versions in 5 Simple Steps

How to Install VirtualBox Guest Additions in Security Onion 14.05

After having to piece together a VirtualBox Guest Addition installation method from some old articles, I decided to put together one for Security Onion 14.05.

Once you have Security Onion 14.05 installed and running, start the Terminal Emulator and walk through the following commands, one-by-one:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo shutdown -r now

After the reboots, launch the Terminal Emulator and run the following commands:

sudo apt-get install build-essential module-assistant dkms
sudo add-apt-repository universe
sudo apt update
sudo m-a prepare

In the virtual machine window, select Devices > Insert Guest Additions CD image

cd /media/%username%/VBOXADDITIONS (tab complete)
sudo ./VBoxLinuxAdditions.run
sudo shutdown -r now

After the reboot, the VirtualBox Guest Additions should be installed and all features should be available.

As a last step, select Devices > Optical Drives > Remove disk from virtual drive…