Tuesday, October 20, 2015

Oracle Enterprise Manager - This report has saved copies

When using Oracle Enterprise Manager to manage your IT footprint you most likely also want to make use of the reporting functions within Oracle Enterprise Manager. Within the latest releases Oracle tries to push to using Oracle BI and not the older reporting options. However, many deployments still use the "old" method of reporting (which works fine in most cases).

In some cases you do want to make a change to a report you have created and might run into a message like this: "You have chosen to edit report "xxxx". This report has saved copies. Do you want to edit the report with limited editing capabilities?".

This means that you cannot change the definition of the report while there are still "old" copies. To resolve this you have to first remove the copies before you can do your changes. To do so, login as a user who has the rights to change the report and open the report itself (not in edit mode, open it in view mode). You will see, as shown in the below screenshot, the number of saved copies.

When you click on the number you will be guided to a page like the one below:

You will have to delete all saved copies of this report. When you have done so and you enter the edit mode of the report again you will see that you have full editing capabilities and are able to make all changes required. 

Thursday, October 08, 2015

Oracle Linux - NuPIC AI core installation

NuPIC is an open source project based on a theory of neocortex called Hierarchical Temporal Memory (HTM). Parts of HTM theory have been implemented, tested, and used in applications, and other parts of HTM theory are still being developed. Today the HTM code in NuPIC can be used to analyze streaming data. It learns the time-based patterns in data, predicts future values, and detects anomalies. HTM is a set of algorithms which model the functionality of the neocortex in the human brain. HTM Theory is the key to unlocking Intelligent Applications and Machines. NupIC is the core product from numenta and is opensource and available to all who like to test with it, build upon it or add to it.

For intelligent applications NuPIC is great as a starting point of your development. However, a thing to keep in mind is that this field of computer science is new, HTM is fairly new. Or in the words from Jeff Hawkins: "This stuff is not easy. I can assure you that once you understand it, you will see a beauty in it. But most people take months to deeply understand the CLA. The tasks of creating hierarchies of CLAs and adding in motor capabilities are very difficult. Even just using the CLA in its current form is not trivial due to the learning required."

When you like to run NuPic on Oracle Linux a number of steps might be a bit different from the installation on a MacBook. Also a couple of dependencies are in place before you can install NuPic on Oracle Linux which are: Python 2.7, Python development headers, pip, wheel, numpy and C++ compiler like gcc or clang.

Python development headers
Next to Python, which most likely will be shipping with your Oracle Linux installation you will have to make sure that you have the Python development headers. You can check if this is installed by executing the below command. In my case Python development headers was already installed.

[root@localhost ~]# rpm -qa | grep python-devel
[root@localhost ~]#

In case you do not get a result you will have to install the Python development headers by executing a yum install command as shown below:

[root@localhost ~]# yum install python-devel

one of the requirements to be able to install NupiC is to install pip. pip is a package management system used to install and manage software packages written in Python. Many packages can be found in the Python Package Index (PyPI). If you have installed the Python setuptools which I describe in this blogpost the installation of pip can be done by using the easy_install command which is part of the setuptools distribution.

[root@localhost ~]# easy_install pip
Searching for pip
Best match: pip 6.1.1
Adding pip 6.1.1 to easy-install.pth file
Installing pip script to /usr/bin
Installing pip3.4 script to /usr/bin
Installing pip3 script to /usr/bin

Using /usr/lib/python2.7/site-packages
Processing dependencies for pip
Finished processing dependencies for pip
[root@localhost ~]#

wheel is required as a dependency. Wheel(s) are a built-package format for Python. A wheel is a ZIP-format archive with a specially formatted filename and the .whl extension. It is designed to contain all the files for a PEP 376 compatible install in a way that is very close to the on-disk format. Many packages will be properly installed with only the “Unpack” step (simply extracting the file onto sys.path), and the unpacked archive preserves enough information to “Spread” (copy data and scripts to their final locations) at any later time. You can install wheel with the just installed pip by executing the below command. Which resulted in my case in some warnings which you can (should) resolve however are not blocking the installation.

[root@localhost ~]# pip install wheel
/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
You are using pip version 6.1.1, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting wheel
/usr/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
    100% |ââââââââââââââââââââââââââââââââ| 65kB 1.8MB/s
Installing collected packages: wheel
Successfully installed wheel-0.26.0
[root@localhost ~]#

it will not come as a surprise that NumPy is required to be installed on the system. NumPy is is the fundamental package for scientific computing with Python. It contains among other things; a powerful N-dimensional array object, sophisticated (broadcasting) functions, tools for integrating C/C++ and Fortran code, useful linear algebra, Fourier transform, and random number capabilities. Besides its obvious scientific uses, NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases

Numpy can be installed by executing the command:
pip install numpy

As the core of Nupic is written in C++ you will need a C++ compiler. The obvious choice in this is GCC which most likely is already installed on your system. You can check the availability with the below command, which shows in my example that it is installed.

[root@localhost ~]# rpm -qa | grep gcc
[root@localhost ~]#

In case it is not installed you can execute a yum install command to install gcc on your Oracle Linux machine. One small, however important note, gcc should be GCC 4.8.

Installing NuPic
After you ensured all dependencies are done you can install NuPic. The installation of NuPic on Oracle Linux is a bit different than the installation on for example a Mac. Reason for this is that The nupic.bindings binary distribution is not stored on PyPi along with the OS X distribution. NuPIC uses the wheel binary format, and PyPi does not support hosting Linux wheel files. This forces you to download the wheel file directly from Numenta and not from PyPi.

pip install https://s3-us-west-2.amazonaws.com/artifacts.numenta.org/numenta/nupic.core/releases/nupic.bindings/nupic.bindings-0.2.1-cp27-none-linux_x86_64.whl
pip install nupic

If all is ok the "pip install nupic" command should work like a charm. However, in case you run into a compiler error like the one shown below it might be that you are missing some additional prerequisites.

cc -c /tmp/tmphmvPkY/vers.cpp -o tmp/tmphmvPkY/vers.o --std=c++11
    cc: error trying to exec 'cc1plus': execvp: No such file or directory
    *WARNING* no libcapnp detected. Will download and build it from source now. If you have C++ Cap'n Proto installed, it may be out of date or is not being detected. Downloading and building libcapnp may take a while.
    fetching https://capnproto.org/capnproto-c++- into /tmp/pip-build-PHQZgs/pycapnp/bundled
    configure: error: *** A compiler with support for C++11 language features is required.

To resolve this issue you will need to do a additional install of gcc-c++ by executing

[root@localhost ~]# yum install gcc-c++

Testing NuPic
to ensure your installation of Nupic was successful you can run a test with the test units provided in the github repository. Execute the py.test against test/unit/ which can be found in the github repository. This should look like the example below.

[root@localhost nupic-master]# py.test tests/unit/
=== test session starts  ===
platform linux2 -- Python 2.7.5 -- pytest-2.5.1
plugins: cov, xdist
collected 844 items / 2 skipped

tests/unit/nupic/utils_test.py ......
tests/unit/nupic/algorithms/anomaly_likelihood_jeff_test.py ...ss..
tests/unit/nupic/algorithms/anomaly_likelihood_test.py ....................
tests/unit/nupic/algorithms/anomaly_test.py ..............
tests/unit/nupic/algorithms/cells4_test.py .
tests/unit/nupic/algorithms/cla_classifier_diff_test.py ...................
tests/unit/nupic/algorithms/cla_classifier_test.py ...................
tests/unit/nupic/algorithms/fast_cla_classifier_test.py ...................
tests/unit/nupic/algorithms/knn_classifier_test.py .....s
tests/unit/nupic/algorithms/nab_detector_test.py ..
tests/unit/nupic/algorithms/sp_overlap_test.py .s.s
tests/unit/nupic/algorithms/svm_test.py ..s
tests/unit/nupic/algorithms/tp10x2_test.py .
tests/unit/nupic/data/aggregator_test.py .
tests/unit/nupic/data/dictutils_test.py ......
tests/unit/nupic/data/fieldmeta_test.py .....
tests/unit/nupic/data/file_record_stream_test.py ......
tests/unit/nupic/data/filters_test.py s
tests/unit/nupic/data/functionsource_test.py ......
tests/unit/nupic/data/inference_shifter_test.py ........
tests/unit/nupic/data/record_stream_test.py .......
tests/unit/nupic/data/utils_test.py .......
tests/unit/nupic/data/generators/anomalyzer_test.py ...........
tests/unit/nupic/data/generators/pattern_machine_test.py .........
tests/unit/nupic/data/generators/sequence_machine_test.py .....
tests/unit/nupic/encoders/adaptivescalar_test.py .......
tests/unit/nupic/encoders/category_test.py ..
tests/unit/nupic/encoders/coordinate_test.py ................
tests/unit/nupic/encoders/date_test.py ........
tests/unit/nupic/encoders/delta_test.py .....
tests/unit/nupic/encoders/geospatial_coordinate_test.py ...........
tests/unit/nupic/encoders/logenc_test.py ......
tests/unit/nupic/encoders/multi_test.py ..
tests/unit/nupic/encoders/pass_through_encoder_test.py ....
tests/unit/nupic/encoders/random_distributed_scalar_test.py ...............
tests/unit/nupic/encoders/scalar_test.py .............
tests/unit/nupic/encoders/scalarspace_test.py .
tests/unit/nupic/encoders/sdrcategory_test.py ...
tests/unit/nupic/encoders/sparse_pass_through_encoder_test.py ....
tests/unit/nupic/engine/network_test.py .........
tests/unit/nupic/engine/syntactic_sugar_test.py .....
tests/unit/nupic/engine/unified_py_parameter_test.py ..
tests/unit/nupic/frameworks/opf/clamodel_classifier_helper_test.py ......................
tests/unit/nupic/frameworks/opf/clamodel_test.py ......
tests/unit/nupic/frameworks/opf/opf_metrics_test.py ...............................
tests/unit/nupic/frameworks/opf/previous_value_model_test.py ......
tests/unit/nupic/frameworks/opf/safe_interpreter_test.py ........
tests/unit/nupic/frameworks/opf/two_gram_model_test.py .....
tests/unit/nupic/frameworks/opf/common_models/cluster_params_test.py .
tests/unit/nupic/math/array_algorithms_test.py ...
tests/unit/nupic/math/cast_mode_test.py s
tests/unit/nupic/math/lgamma_test.py .
tests/unit/nupic/math/nupic_random_test.py .............
tests/unit/nupic/math/sparse_binary_matrix_test.py ............s............
tests/unit/nupic/math/sparse_matrix_test.py ...s...............................
tests/unit/nupic/regions/anomaly_region_test.py .
tests/unit/nupic/regions/knn_anomaly_classifier_region_test.py ....................
tests/unit/nupic/regions/pyregion_test.py ....
tests/unit/nupic/regions/record_sensor_region_test.py .
tests/unit/nupic/regions/regions_spec_test.py s...s......
tests/unit/nupic/research/connections_test.py .............
tests/unit/nupic/research/inhibition_object_test.py s
tests/unit/nupic/research/sp_learn_inference_test.py s
tests/unit/nupic/research/spatial_pooler_boost_test.py ..
tests/unit/nupic/research/spatial_pooler_compatability_test.py ....ss..
tests/unit/nupic/research/spatial_pooler_compute_test.py ..
tests/unit/nupic/research/spatial_pooler_cpp_api_test.py ..............................
tests/unit/nupic/research/spatial_pooler_py_api_test.py ..............................
tests/unit/nupic/research/spatial_pooler_unit_test.py s.................................
tests/unit/nupic/research/temporal_memory_test.py ...........................
tests/unit/nupic/research/tp10x2_test.py ....
tests/unit/nupic/research/tp_constant_test.py ...
tests/unit/nupic/research/tp_test.py ....
tests/unit/nupic/research/monitor_mixin/metric_test.py ..
tests/unit/nupic/research/monitor_mixin/trace_test.py ..
tests/unit/nupic/support/configuration_test.py ............s....................
tests/unit/nupic/support/custom_configuration_test.py .........s..............
tests/unit/nupic/support/decorators_test.py ....
tests/unit/nupic/support/object_json_test.py ...............
tests/unit/nupic/support/consoleprinter_test/consoleprinter_test.py .
=== 825 passed, 21 skipped in 100.95 seconds ===
[root@localhost nupic-master]#

This should enable you to start with exploring NuPic

Tuesday, October 06, 2015

Oracle Linux - Install Python setuptools

When working with Python and when you like to make your life more easy when installing new modules and functions it is commonly a best practice to use things like for example pip and/pr Python setuptools. Python setuptools will help you to easily download, build, install, upgrade, and uninstall Python packages. The setup of the setuptools on Oracle Linux is basically a single command to get things working. Executing the command will download a python script and execute it. This script will ensure the setuptool will be downloaded and installed correctly on your system.

You can download and execute the script manually and in two steps, you can also do this in one go and ensure that you only need a single command to install the setuptools on Oracle Linux. Below is an example of the single command which involves a wget and sending the result to Python for execution.

[root@localhost ~]# wget https://bootstrap.pypa.io/ez_setup.py -O - | python
--2015-10-06 16:06:27--  https://bootstrap.pypa.io/ez_setup.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)...
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)||:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 11434 (11K) [text/x-python]
Saving to: âSTDOUTâ

100%[==================================>] 11,434      --.-K/s   in 0s

2015-10-06 16:06:28 (534 MB/s) - written to stdout [11434/11434]

Downloading https://pypi.python.org/packages/source/s/setuptools/setuptools-18.3.2.zip
Extracting in /tmp/tmpuwKkuT
Now working in /tmp/tmpuwKkuT/setuptools-18.3.2
Installing Setuptools
running install
running bdist_egg
running egg_info
writing requirements to setuptools.egg-info/requires.txt
writing setuptools.egg-info/PKG-INFO
writing top-level names to setuptools.egg-info/top_level.txt
writing dependency_links to setuptools.egg-info/dependency_links.txt
writing entry points to setuptools.egg-info/entry_points.txt
reading manifest file 'setuptools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'setuptools.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
copying easy_install.py -> build/lib
creating build/lib/_markerlib
copying _markerlib/__init__.py -> build/lib/_markerlib
copying _markerlib/markers.py -> build/lib/_markerlib
creating build/lib/pkg_resources
copying pkg_resources/__init__.py -> build/lib/pkg_resources
creating build/lib/setuptools
copying setuptools/__init__.py -> build/lib/setuptools
copying setuptools/archive_util.py -> build/lib/setuptools
copying setuptools/compat.py -> build/lib/setuptools
copying setuptools/depends.py -> build/lib/setuptools
copying setuptools/dist.py -> build/lib/setuptools
copying setuptools/extension.py -> build/lib/setuptools
copying setuptools/lib2to3_ex.py -> build/lib/setuptools
copying setuptools/msvc9_support.py -> build/lib/setuptools
copying setuptools/package_index.py -> build/lib/setuptools
copying setuptools/py26compat.py -> build/lib/setuptools
copying setuptools/py27compat.py -> build/lib/setuptools
copying setuptools/py31compat.py -> build/lib/setuptools
copying setuptools/sandbox.py -> build/lib/setuptools
copying setuptools/site-patch.py -> build/lib/setuptools
copying setuptools/ssl_support.py -> build/lib/setuptools
copying setuptools/unicode_utils.py -> build/lib/setuptools
copying setuptools/utils.py -> build/lib/setuptools
copying setuptools/version.py -> build/lib/setuptools
copying setuptools/windows_support.py -> build/lib/setuptools
creating build/lib/pkg_resources/_vendor
copying pkg_resources/_vendor/__init__.py -> build/lib/pkg_resources/_vendor
creating build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/__about__.py -> build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/__init__.py -> build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/_compat.py -> build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/_structures.py -> build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/specifiers.py -> build/lib/pkg_resources/_vendor/packaging
copying pkg_resources/_vendor/packaging/version.py -> build/lib/pkg_resources/_vendor/packaging
creating build/lib/setuptools/command
copying setuptools/command/__init__.py -> build/lib/setuptools/command
copying setuptools/command/alias.py -> build/lib/setuptools/command
copying setuptools/command/bdist_egg.py -> build/lib/setuptools/command
copying setuptools/command/bdist_rpm.py -> build/lib/setuptools/command
copying setuptools/command/bdist_wininst.py -> build/lib/setuptools/command
copying setuptools/command/build_ext.py -> build/lib/setuptools/command
copying setuptools/command/build_py.py -> build/lib/setuptools/command
copying setuptools/command/develop.py -> build/lib/setuptools/command
copying setuptools/command/easy_install.py -> build/lib/setuptools/command
copying setuptools/command/egg_info.py -> build/lib/setuptools/command
copying setuptools/command/install.py -> build/lib/setuptools/command
copying setuptools/command/install_egg_info.py -> build/lib/setuptools/command
copying setuptools/command/install_lib.py -> build/lib/setuptools/command
copying setuptools/command/install_scripts.py -> build/lib/setuptools/command
copying setuptools/command/register.py -> build/lib/setuptools/command
copying setuptools/command/rotate.py -> build/lib/setuptools/command
copying setuptools/command/saveopts.py -> build/lib/setuptools/command
copying setuptools/command/sdist.py -> build/lib/setuptools/command
copying setuptools/command/setopt.py -> build/lib/setuptools/command
copying setuptools/command/test.py -> build/lib/setuptools/command
copying setuptools/command/upload_docs.py -> build/lib/setuptools/command
copying setuptools/script (dev).tmpl -> build/lib/setuptools
copying setuptools/script.tmpl -> build/lib/setuptools
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib/easy_install.py -> build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/_markerlib
copying build/lib/_markerlib/__init__.py -> build/bdist.linux-x86_64/egg/_markerlib
copying build/lib/_markerlib/markers.py -> build/bdist.linux-x86_64/egg/_markerlib
creating build/bdist.linux-x86_64/egg/pkg_resources
copying build/lib/pkg_resources/__init__.py -> build/bdist.linux-x86_64/egg/pkg_resources
creating build/bdist.linux-x86_64/egg/pkg_resources/_vendor
copying build/lib/pkg_resources/_vendor/__init__.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor
creating build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/__about__.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/__init__.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/_compat.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/_structures.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/specifiers.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
copying build/lib/pkg_resources/_vendor/packaging/version.py -> build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging
creating build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/__init__.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/archive_util.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/compat.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/depends.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/dist.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/extension.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/lib2to3_ex.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/msvc9_support.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/package_index.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/py26compat.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/py27compat.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/py31compat.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/sandbox.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/site-patch.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/ssl_support.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/unicode_utils.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/utils.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/version.py -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/windows_support.py -> build/bdist.linux-x86_64/egg/setuptools
creating build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/__init__.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/alias.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/bdist_egg.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/bdist_rpm.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/bdist_wininst.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/build_ext.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/build_py.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/develop.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/easy_install.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/egg_info.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/install.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/install_egg_info.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/install_lib.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/install_scripts.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/register.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/rotate.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/saveopts.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/sdist.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/setopt.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/test.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/command/upload_docs.py -> build/bdist.linux-x86_64/egg/setuptools/command
copying build/lib/setuptools/script (dev).tmpl -> build/bdist.linux-x86_64/egg/setuptools
copying build/lib/setuptools/script.tmpl -> build/bdist.linux-x86_64/egg/setuptools
byte-compiling build/bdist.linux-x86_64/egg/easy_install.py to easy_install.pyc
byte-compiling build/bdist.linux-x86_64/egg/_markerlib/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/_markerlib/markers.py to markers.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/__about__.py to __about__.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/_compat.py to _compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/_structures.py to _structures.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/specifiers.py to specifiers.pyc
byte-compiling build/bdist.linux-x86_64/egg/pkg_resources/_vendor/packaging/version.py to version.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/archive_util.py to archive_util.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/compat.py to compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/depends.py to depends.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/dist.py to dist.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/extension.py to extension.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/lib2to3_ex.py to lib2to3_ex.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/msvc9_support.py to msvc9_support.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/package_index.py to package_index.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/py26compat.py to py26compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/py27compat.py to py27compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/py31compat.py to py31compat.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/sandbox.py to sandbox.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/site-patch.py to site-patch.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/ssl_support.py to ssl_support.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/unicode_utils.py to unicode_utils.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/utils.py to utils.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/version.py to version.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/windows_support.py to windows_support.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/alias.py to alias.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_egg.py to bdist_egg.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_rpm.py to bdist_rpm.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/bdist_wininst.py to bdist_wininst.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/build_ext.py to build_ext.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/build_py.py to build_py.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/develop.py to develop.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/easy_install.py to easy_install.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/egg_info.py to egg_info.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install.py to install.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_egg_info.py to install_egg_info.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_lib.py to install_lib.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/install_scripts.py to install_scripts.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/register.py to register.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/rotate.py to rotate.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/saveopts.py to saveopts.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/sdist.py to sdist.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/setopt.py to setopt.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/test.py to test.pyc
byte-compiling build/bdist.linux-x86_64/egg/setuptools/command/upload_docs.py to upload_docs.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying setuptools.egg-info/zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
creating dist
creating 'dist/setuptools-18.3.2-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing setuptools-18.3.2-py2.7.egg
Copying setuptools-18.3.2-py2.7.egg to /usr/lib/python2.7/site-packages
Adding setuptools 18.3.2 to easy-install.pth file
Installing easy_install script to /usr/bin
Installing easy_install-2.7 script to /usr/bin

Installed /usr/lib/python2.7/site-packages/setuptools-18.3.2-py2.7.egg
Processing dependencies for setuptools==18.3.2
Finished processing dependencies for setuptools==18.3.2
[root@localhost ~]#

In esscence there is nothing more to installing the Python setuptools on Oracle Linux. A single command will ensure you are in business and you are good to go. 

Oracle Linux - generate MAC address

In most cases you will not need to generate a MAC address. It will come with your network interface or, in cases of a virtual machine, it will be generated for you by the orchestration tooling. However, in some cases you might need to generate a random MAC address. In my case this was when we experimented with the Oracle VM API's and at some point in time we wanted to provide the MAC address to the code that orchestrated the creation and deployment of a new VM.

Generating a new MAC address can be done in multiple ways, the below Python script is just one of the examples, however, it can be intergrated faitly easy into wider Python code or you can call it from a Bash script.

# macgen.py script to generate a MAC address for guests on Xen
import random
def randomMAC():
 mac = [ 0x00, 0x16, 0x3e,
  random.randint(0x00, 0x7f),
  random.randint(0x00, 0xff),
  random.randint(0x00, 0xff) ]
 return ':'.join(map(lambda x: "%02x" % x, mac))
print randomMAC()

This script will simply provide you a complete random MAC address.

Oracle VM - selecting a DR layer

When developing a new solution, including both the application, data-store and infrastructure components, one of the questions to ask is on which layer to build resilience against failure. On which level of the stack will you protect against failure of a component and on which level will your disaster recovery focus. In essence the answer is quite simple, you should ensure that disaster recovery is safeguarded as high as possible in the stack. The true answer is a very complex answer which includes disaster recovery, high availability and maximum availability components. Building a solution which is resilient against failure is a very complex process in which every component needs to be taken into account. However, making sure that you have disaster recovery as high up in the stack as possible will make your life much more easy.

As an example we take the below image which shows a application centered disaster recovery solution based within a virtualized environment with Oracle VM.

Within this solution applications will run in a active active setup in both site A as well as site B. Information between the two sites is kept in sync by making use of the MAA maximum Availability Architecture principles from Oracle. This means that when a site fails the application will still be able to function as it will run on the other site. Users should not face any downtime and should not even be aware that one of the two sites has been lost due to a disaster.

The application centered disaster recovery solution is the most resilient solution against disasters and the loss of a site. However, in some cases it is not feasible to run a architecture as shown above and you would still like to be able to perform a disaster recovery of the virtual machines running within your deployment. A solution to this is making use of block replication on a storage level and allowing your recovery site (site B) to start the VM's in case your site A is lost.

Within this model you will replicate all storage associated with the VM's from site A to a storage repository within site B. In essence this is an exact copy of the VM, however, on site B the machine is in a stopped state. This is also represented in the diagram below where you can more clearly see the replication of storage on the two sides. For this solution you can use storage block replication in a way that your storage appliance is supporting.

In case of a failure you have to ensure that all machines are stopped on site A, after this you can make the storage on site B readable and writable and start the virtual machines. This might not be the most ideal solution in comparison with disaster recovery in the higher levels of the stack, however, in case you are forced to ensure disaster recovery on a infrastructure / VM layer instead of a application level this is a solution that can be used. 

For more information, also view the slidedeck below.

Friday, October 02, 2015

Oracle Linux - detect security issues

When operating a large landscape of Linux machines, in our case a large landscape of Oracle Linux machines security is one of the vital things to keep in mind. In an ideal world all your Linux deployments would be of exactly the same version and contain exactly the same level of patching. In an ideal world no machine would differ from another machine and in this same ideal world you would be able to run a yum update command on all machines and would never face any issue nor would you be required to talk to end-customers or other tech team. However, even though in some situations you are able to maintain such a situation, commonly it is seen that a landscape of servers is equally patched and in some cases servers are not patched for a long period of time. This is not necessarily due to bad maintenance by the Linux administrators, commonly it is related to pressure from the business not to change the systems or not getting approval from a change advisory board.

When it comes down to new or improved functionality which can come with a Linux patch this might be acceptable. However, in case of missing a security patch this might be much more serious. Oracle Enterprise Manager provides, in combination with Yum a solution to show which patches need to be applied on which system. However, also a different solution can be used specifically to identify which security issues have not been addressed in a specific system.

To get an overview of which security vulnerabilities are on your system you can use OpenSCAP. OpenSCAP is based upon SCAP is a line of standards managed by NIST. It was created to provide a standardized approach to maintaining the security of enterprise systems, such as automatically verifying the presence of patches, checking system security configuration settings, and examining systems for signs of compromise.

Oracle provides a OVAL®: Open Vulnerability and Assessment Language XML file which you can use in combination with OpenSCAP to run against your Oracle Linux deployement to get a quick overview of what needs attention on your system and what looks to be correct. If you refer to the Oracle Linux security guide you can find more information around this subject.

After you have installed the needed components with using a Yum command you will have to download the Oracle Linux specific components, or in more detail, the Oracle Linux ELSA file in OVAL format. Oracle provides this file in year files where each year file contains the information on security issues found during that year. As an example, if you wanted to run an audit against the ELSA file of 2015 you need to perform the following steps:

1) Download the ELSA information in the OVAL format and extract it from the bz2 file
wget http://linux.oracle.com/security/oval/com.oracle.elsa-2015.xml.bz2
bzip2 -d com.oracle.elsa-2015.xml.bz2

2) Run the audit. In this case we send both the XML result as well as the HTML report to /tmp however you are free to select any location you want.
oscap oval eval --results /tmp/elsa-results-oval-2015.xml --report /tmp/elsa-report-2015.html ./com.oracle.elsa-2015.xml

This will produce a rather large output to the screen which provides some quick information however the more valuable information can be found in both the XML result as well as in the HTML report which we have send to /tmp . For references the below is the shell output of the audit on the 2015 file which I ran against a Oracle Linux 3.8.13-98.2.2.el7uek.x86_64 implementation:
[root@localhost oscap]# oscap oval eval --results /tmp/elsa-results-oval-2015.xml --report /tmp/elsa-report-2015.html ./com.oracle.elsa-2015.xml
Definition oval:com.oracle.elsa:def:20153073: false
Definition oval:com.oracle.elsa:def:20153072: false
Definition oval:com.oracle.elsa:def:20153071: true

//------------ SNIP SNIP ------------//

Definition oval:com.oracle.elsa:def:20150166: false
Definition oval:com.oracle.elsa:def:20150165: false
Definition oval:com.oracle.elsa:def:201501641: false
Definition oval:com.oracle.elsa:def:20150164: false
Definition oval:com.oracle.elsa:def:20150118: false
Definition oval:com.oracle.elsa:def:20150102: true
Definition oval:com.oracle.elsa:def:20150100: false
Definition oval:com.oracle.elsa:def:20150092: false
Definition oval:com.oracle.elsa:def:20150090: false
Definition oval:com.oracle.elsa:def:20150087: false
Definition oval:com.oracle.elsa:def:20150085: false
Definition oval:com.oracle.elsa:def:20150074: false
Definition oval:com.oracle.elsa:def:20150069: false
Definition oval:com.oracle.elsa:def:20150068: false
Definition oval:com.oracle.elsa:def:20150067: false
Definition oval:com.oracle.elsa:def:20150066: false
Definition oval:com.oracle.elsa:def:20150047: false
Definition oval:com.oracle.elsa:def:20150046: false
Definition oval:com.oracle.elsa:def:20150016: false
Definition oval:com.oracle.elsa:def:20150008: false
Evaluation done.
[root@localhost oscap]# 

3) Review the results (and take action)
You will have to review the results, which can be done by looking at the HTML report or you can run a parser against the XML output to do a more automated way of checking the results. In case you run a large number of Oracle Linux machines and you like to use the oscap way of checking parts of your security you most likely want to have the xml files somewhere in a central location so you do not need to connect all your machines to the public internet and you most likley want to run this in a scheduled form and interpret the results in a automated manner. The HTML file is usable for human reading, however, the XML file is something you would like to parse and use in case you have more then x servers.