Friday, October 14, 2016

Using osquery in Oracle Linux

Recently the guys at facebook released an internal project as opensource code. Now you can make use of some of the internal solutions facebook is using to keep track and analyse their compute nodes in the facebook datacenter. Osquery allows you to easily ask questions about your Linux, Windows, and OS X infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and inform a broad set of organizations within your company.

What osquery provides is a collector that on a scheduled basis will analyse your operating system and store this information in a sqlite database local on your system. In essence osquery is an easily configurable and extensible framework that will do the majority of collection tasks for you. What makes it a great product is that it is all stored in sqlite and that enables you to use standard SQL code to ask questions about your system.

After a headsup from Oracle Linux product teams about the fact that facebook released this as opensource I installed it on an Oracle Linux instance to investigate the usability of osquery.

Installing osquery
Installation is quite straightforward. A RPM is provided which installs without any issue on Oracle Linux 6. Below is an example of downloading and installing osquery on an Oracle Linux 6 instance.

[root@testbox08 ~]#
[root@testbox08 ~]# wget "https://osquery-packages.s3.amazonaws.com/centos6/osquery-2.0.0.rpm" -b
Continuing in background, pid 28491.
Output will be written to “wget-log”.
[root@testbox08 ~]#
[root@testbox08 ~]# ls -rtl osq*.rpm
-rw-r--r-- 1 root root 13671146 Oct  4 17:13 osquery-2.0.0.rpm
[root@testbox08 ~]# rpm -ivh osquery-2.0.0.rpm
warning: osquery-2.0.0.rpm: Header V4 RSA/SHA256 Signature, key ID c9d8b80b: NOKEY
Preparing...                ########################################### [100%]
   1:osquery                ########################################### [100%]
[root@testbox08 ~]#
[root@testbox08 ~]#

When you check you will notice that osquery will not start by default and that some manual actions are required to get it started. In essence this is due to the fact that no default configuration is provided during the installation. To enable the collector (daemon) to start it will look for the configuration file /etc/osquery/osquery.conf to be available. This is not a file that is part of the RPM installation. This will result in the below warning when you try to start the osquery daemon;

[root@testbox08 init.d]#
[root@testbox08 init.d]# ./osqueryd start
No config file found at /etc/osquery/osquery.conf
Additionally, no flags file or config override found at /etc/osquery/osquery.flags
See '/usr/share/osquery/osquery.example.conf' for an example config.
[root@testbox08 init.d]#

Without going into the details of how to configure osquery and tune it for you specific installation you can start to test osquery by simply using the default example configuration file.

[root@testbox08 osquery]#
[root@testbox08 osquery]# cp /usr/share/osquery/osquery.example.conf /etc/osquery/osquery.conf
[root@testbox08 osquery]# cd /etc/init.d
[root@testbox08 init.d]# ./osqueryd start
[root@testbox08 init.d]# ./osqueryd status
osqueryd is already running: 28514
[root@testbox08 init.d]#
[root@testbox08 osquery]#

As you can see, we now have the osquery deamon osqueryd running under PID 28514. As it is a collector it is good to wait for a couple of seconds to ensure the collector makes its first collection and stores this in the sqlite database. However, as soon as it has done so you should be able to get the first results stored in your database and you should be able to query the results for data.

To make life more easy, you can use the below script to install osquery in a single go:

#!/bin/sh
wget "https://osquery-packages.s3.amazonaws.com/centos6/osquery-2.0.0.rpm" -O /tmp/osquery.rpm
rpm -ivh /tmp/osquery.rpm
rm -f /tmp/osquery.rpm
cp /usr/share/osquery/osquery.example.conf /etc/osquery/osquery.conf
./etc/init.d/osqueryd start

Using osqueryi
The main way to interact with the osquery data is using osqueryi which is located at /usr/bin/osqueryi . Which means that if you execute osqueryi you will be presented a command line interface you can use to query the data collected by the osqueryd collector. 

[root@testbox08 /]#
[root@testbox08 /]# osqueryi
osquery - being built, with love, at Facebook
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Using a virtual database. Need help, type '.help'
osquery>

As an example you can query which pci devices are present with a single SQL query as shown below:

osquery>
select * from pci_devices;
+--------------+-----------+------------------+--------+-----------+-------+----------+
| pci_slot     | pci_class | driver           | vendor | vendor_id | model | model_id |
+--------------+-----------+------------------+--------+-----------+-------+----------+
| 0000:00:00.0 |           |                  |        | 8086      |       | 1237     |
| 0000:00:01.0 |           |                  |        | 8086      |       | 7000     |
| 0000:00:01.1 |           | ata_piix         |        | 8086      |       | 7010     |
| 0000:00:01.3 |           |                  |        | 8086      |       | 7113     |
| 0000:00:02.0 |           |                  |        | 1013      |       | 00B8     |
| 0000:00:03.0 |           | xen-platform-pci |        | 5853      |       | 0001     |
+--------------+-----------+------------------+--------+-----------+-------+----------+
osquery>

As osqueryi uses a sqlite backend we can use the standard options and SQL provided by sqlite and for example get a full overview of all tables that are present when using the .table command in the command line interface. This provides the below output, which can be a good start to investigate what type of information is being collected by default and can be used;

  acpi_tables
  apt_sources
  arp_cache
  authorized_keys
  block_devices
  carbon_black_info
  chrome_extensions
  cpu_time
  cpuid
  crontab
  deb_packages
  device_file
  device_hash
  device_partitions
  disk_encryption
  dns_resolvers
  etc_hosts
  etc_protocols
  etc_services
  file
  file_events
  firefox_addons
  groups
  hardware_events
  hash
  interface_addresses
  interface_details
  iptables
  kernel_info
  kernel_integrity
  kernel_modules
  known_hosts
  last
  listening_ports
  logged_in_users
  magic
  memory_info
  memory_map
  mounts
  msr
  opera_extensions
  os_version
  osquery_events
  osquery_extensions
  osquery_flags
  osquery_info
  osquery_packs
  osquery_registry
  osquery_schedule
  pci_devices
  platform_info
  process_envs
  process_events
  process_memory_map
  process_open_files
  process_open_sockets
  processes
  routes
  rpm_package_files
  rpm_packages
  shared_memory
  shell_history
  smbios_tables
  socket_events
  suid_bin
  syslog
  system_controls
  system_info
  time
  uptime
  usb_devices
  user_events
  user_groups
  user_ssh_keys
  users
  yara
  yara_events

The example shown above is a extreme simple example, everyone with at least a bit SQL experience will be able to write much more extensive and interesting queries which can make life as a Linux administrator much more easy.

Script against osquery
Even though using the command line interface is nice for adhoc queries you might have for a single Oracle Linux instance it is more interesting to see how you can use osquery in a scripted manner. As this is based upon sqlite you can use the same solutions you would use when coding against a standard sqlite database. This means you can use bash scripting, however, you can also use most other scripting languages and programming languages popular on the Linux platform. Most languages now have options to interact with a sqlite database. 

No comments: