May 312012

Selinux works pretty well straight out of the box, but I have learnt that when application reconfigurations mysteriously result in failures to startup – then Selinux is generally the culprit. In particular, if any file locations are customised from what’s installed, Selinux will lock it down, and for the uninitiated, the debugging process can be quite confusing. Apache webserver (HTTPD) provides a good example of what happens in this scenario, and I’ll demonstrate how to make it work with the Document Root in a different location.

Selinux can be distinctly daunting for Linux sysadmins, and often seems to be more trouble than it’s worth. However it is an extremely powerful tool which increases security and encourages rigour in configuration. Essentially, Selinux augments the usual Unix file permissions and ownership by adding more granular application-specific contexts as well. I won’t go into the details here, as it’s well documented elsewhere.

What I’ll aim to show here is two ways for relocating the Apache DocumentRoot directory – the proper way, and a quicker dirtier way.

Say, you decide to situate the Apache webserver root under /u01/www/html instead of the default /var/www/html. You edit the /etc/httpd/conf/httpd.conf and change the DocumentRoot line to:

DocumentRoot /u01/www/html

Copy your HTML files into place, change the directory and file onwerships, and permissions:

# chown -R apache. /u01/www/html
# chmod 755 /u01/www/html
# chmod 644 /u01/www/html/*

(NB: That’s not a typo – you can type “apache.”, with the dot at the end, in chown rather than “apache:apache” and save yourself a few keystrokes).

Restart httpd to make sure, and browse to your web page. You’ll get something like this in your browser:

You don't have permission to access / on this server.

And in /var/log/httpd/error_log:

[client] (13)Permission denied: access to /index.html denied

This is because the Apache back-end process, running as user “apache”, can’t access the file /u01/www/html/index.html – even though we granted all the correct permissions. What gives? Turns out, with Selinux, having regular old permissions and ownership isn’t enough.

Here’s how you can fix this, nice and quickly.

First things first, check that you’ve got Selinux running, and that this is in fact the source of your problem:

  # getenforce

Yep. Next, check the Selinux context of the original HTTPD DocumentRoot directory. The ls command has the “-Z” switch which displays this:

# ls -ldZ /var/www/html
drwxr-xr-x. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html

Take note of the colon-delimited string “system_u:object_r:httpd_sys_content_t:s0”. Take note of this string; you’ll need it later. Each field is:

  • User: system_u
  • Role: object_r
  • Type: httpd_sys_content_t
  • Level: s0

Checking the Selinux context on our new erroneously configured folder, we see this:

# ls -Z file1
drwxr-xr-x. root root unconfined_u:object_r:default_t:s0 /u01/www

So this explains why Apache can’t read the index.html file – it’s got the wrong Selinux context for Apache to read it. You can see how this enhances security. Apache can never server up a page unless it has explicitly got a context set that matches Apache privilege.

To reset the directory and file contexts, you need to ensure that the semanage software is installed (it may not be, depending on what package groups you have):

  # yum whatprovides */semanage
  # yum -y install policycoreutils-python

A very full discussion of how to change Selinux policies and file contexts is given here, but briefly, the steps are:

  # semanage fcontext -a -t httpd_sys_content_t "/u01/www(/.*)?"
  # restorecon -Rv /u01/www
restorecon reset /u01/www context unconfined_u:object_r:var_t:s0->system_u:object_r:httpd_sys_content_t:s0
restorecon reset /u01/www/html context unconfined_u:object_r:var_t:s0->system_u:object_r:httpd_sys_content_t:s0
restorecon reset /u01/www/html/index.html context unconfined_u:object_r:var_t:s0->system_u:object_r:httpd_sys_content_t:s0

The semanage argument “u01/www(/.*)?” is a regular expression indicating the context should apply to the directory “/u01/www” and everything under it.

Quick and Dirty Fix
If, however you don’t have access to the semanage program or if you just need to make a small ad hoc change to a file context, you can simply set this manually. Remember that string coloured red earlier? Grab the relevant text from it.

  # chcon -R -u system_u -t httpd_sys_content_t /u01/www/

This command uses the “-R” switch so will apply recursively to all files under the directory. Note that the difference between this command and the semanage command is that chcon will not update the Selinux policy.

So the proof of the pudding will be in the reloading. Ctrl-R your browser and BOOM! Your page is back and being happily server up.

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

May 212012

When you start operating Apache HTTPD webserver with a lot of virtual hosts, the configuration can become quite unwieldy and difficult to debug. It can become an unfortunate process of trial and error to work out whether all of your virtual hosts are up and running, and which one is the default (it’s the first one alphabetically, unless specified). In fact, working out the name of your default virtual host is often the key to solving many Apache problems.

But there is a rather nice and quick debugging trick for getting Apache to just dump out a list of all its running virtual hosts. The command line is simply this:

  # httpd -D DUMP_VHOSTS

Which outputs:

VirtualHost configuration: (/etc/httpd/conf.d/secure_website.conf:4)
wildcard NameVirtualHosts and _default_ servers:
*:80                   is a NameVirtualHost
      default server (/etc/httpd/conf.d/website.conf:1)
      port 80 namevhost (/etc/httpd/conf.d/website.conf:1)
      port 80 namevhost (/etc/httpd/conf.d/website.conf:10)
      port 80 namevhost (/etc/httpd/conf.d/website.conf:19)
      port 80 namevhost (/etc/httpd/conf.d/website.conf:29)

Hurrah! That’s a list of the URLs of all virtual hosts, and the conf.d files in which they’re configured.

And better yet, if you run this command with the “-t” switch, it will instead parse the configuration files, and not the running configuration:

  # httpd -t -D DUMP_VHOSTS

So you can test your configuration (note that it also performs a syntax check) before loading it.

Finally, when you’re satisfied that your changes are correct, reload the configuration:

  # apachectl configtest
  # apachectl restart

Or, on Fedora:

  # apachectl configtest
  # service httpd reload

Special bonus Apache debug information!
If you run httpd on the command line with the “-L” switch, you get a list of all recognised directives, their descriptions, and the contexts in which they are allowed. It’s extremely handy to have this information to hand, and to not have to revert to the manual every time.

  # httpd -L
<Directory (core.c)
  Container for directives affecting resources located in the specified directories
  Allowed in *.conf only outside <Directory>, <Files> or <Location>
<Location (core.c)
  Container for directives affecting resources accessed through the specified URL paths
  Allowed in *.conf only outside <Directory>, <Files> or <Location>
<VirtualHost (core.c)
  Container to map directives to a particular virtual host, takes one or more host addresses
  Allowed in *.conf only outside <Directory>, <Files> or <Location>

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

May 172012

I’m sure you’ve done this before. You kick off some process in the background that you know will take a while, but then you realise that you need to logout, and that if you do, then the process will die when the shell exits. What to do? Surely you’ll not have to start again?

So you’ve done this:

   # rsync -a host1:/u01/data/ host2:/u02 &

What’s that? You forgot to background it and type the “&” ? Just type “Ctrl-Z” and then type bg – now it’s in the background.

So you’ve realised that you’ve kicked off a process without nohup and that you’ll need to logout. But you don’t want to lose the work the process has done, and don’t want to start it again. You wish you’d thought to run it with nohup in the first place, like this:

  # nohup rsync -a host1:/u01/data/ host2:/u02 &
[2] 4885
nohup: ignoring input and appending output to `nohup.out'

Is it too late? Do you really need to kill the backgrounded process and start it again from the beginning when it’s already come so far?

Not on your life, because there is a little bash builtin tucked away at the back of the man page that’s easy to miss – disown

All you need to do is execute disown with the job number as an argument. This job control procedure goes something like this (the number “[1]” is the job number):

  # jobs
[1]+ Running       rsync -a host1:/u01/data/ host2:/u02 &
  # disown %1

This will detach the detach the process with job number 1 – the rsync – from the running shell and prevent it from being orphaned when the shell is exited. The rsync will continue to run happily on its own while you’re off in the pub. Problem solved.

You’re welcome.

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.

May 142012

The telnet service is never used these days as a means of accessing a host shell – it’s too insecure – but the telnet client is, for many sysadmins, still the tool of choice for testing whether a host has connectivity with another host network service. It’s a good reliable and general tool, and the syntax is dead simple. So for example, if you wanted to just test whether you could hit the Tomcat port (8080) on host “davros”, you’d:

   # telnet davros 8080

But telnet isn’t really the “right” tool for this job. By default it opens a TCP connection to a nominated port, but this flexible port argument is more of a side effect of its function as an interactive console, and not its real function. So from now on, you should be using netcat – which is executed as nc.


Netcat is a Swiss army knife for network connections, and as well as being able to initiate outgoing calls to TCP and UDP ports, it can be made to function as a listener to read incoming connections. It can be used to construct a basic proxy. Most importantly, because netcat is non-interactive (unlike telnet) it can be easily used in scripts. The connection will be terminated gracefully, rather than left rudely cut off from the prompt with telnet’s “^]”.

Netcat is installed by default on several Linux distributions and is easily obtainable from the standard repositories on others.

Working Example

Here’s a few basic examples which show how it’s used to test network connectivity. Note that extent of the tests is just to check whether the port is listening, and not the nature of the daemon or whether it’s working. In every case, I’ll assume once again that our listening host is called “davros”.

Test that the Tomcat port (tcp/8080) is listening and accessible

   # nc -v davros 8080
Connection to davros 8080 port [tcp/http-alt] succeeded!

The man page has a lot of other good examples which are worth trying out. Netcat is a very versatile, yet very basic command, admirabl suited to creating TCP and UDP sockets.

Matt Parsons is a freelance Linux specialist who has designed, built and supported Unix and Linux systems in the finance, telecommunications and media industries.

He lives and works in London.