Tag Archives: apache

Systemd in the heart of Redhat 7 service management:

If you’re a redhat linux user or you’re using a distribution based on Redhat Linux, you probably wonder what’s in Redhat Linux 7 for you.

7cent

Well RHEL 7 brought a lot of new features, beginning with kernel version…. for a sys admin, the big news is the init system, RHEL is now using systemd as the default init system.

What is Systemd?

Systemd is an init system developed by the engineer Lenart Poettering.

If you don’t know, init system is simply the process with PID 1 responsible for spawning all other processes and bringing up the user space within your system. It’s the program your boat loader point to so the kernel uses it to start everything user space.

btprc

You certainly had heard of init systems like SysVinit and upstart, systemd is much more elegant 😉

 

How it works?

Systemd uses the concept of “Unit” to manage services, their dependencies and the allocated resources.

Think of everything services, swap files, mount points, run levels, devices… can be categorized under units. So each device, service or mount point may have a configuration file .unit_name  containing details like binary file to run a service, dependency with others services, mount points….

Systemd Units are as follows:

service, socket, device, mount, automount, swap, target, path, timer, snapshot, slice and scope.

You remember /etc/init.d/ scripts? So think of unit file as init.d scripts..

Apache service for instance, will have a httpd.service unit file which could be configured as follows:

httunit

Look at the [service] part, it points to executable file to run/reload/stop the service

With this file, you can configure the run level using this service, look at [Install] part and WantedBy keyword, dependency with other services, observe the [Unit] part and the After keyword. So here, multi-user.target requires apache service.

Target units correspond to classic Runlevels, each target file contains details about required services to be launched. You can have multi-user.target, graphical.target….

In apache case mentioned above, observe that multi-user.target requirs apache service. In this case and generally, you’ll find a directory /etc/systemd/system/multi-user.wants/ containing a symbolic link pointing to apache service unit and generally to all required services.

 

What I like the most about systemd :

  • Systemd exploits Cgroups

Systemd uses cgroups kernel to track everything about your processes.

Cgroups are a built-in kernel feature which creates an hierarchy for groups of processes. Think of cgroups as containers that child processes couldn’t escape even with a double fork making it very easy to track them. These containers could serve also for resource management (cpu, ram,…) of a particular process..

If you’re a sys admin, remember how many times you couldn’t determine which service/process spawned a strange process? Or you couldn’t kill all your CGI processes even after shutting down your apache? Well with systemd exploiting cgroups, there is no chance for this to happen J

You can find your cgroups in /sys/fs/cgroups

cgroup

The command systemd-cgtop returns the top cgroups defined in your system, ordered by their CPU,RAM and load:

systemd-cgtop

  •  Systemd’s journal:

Systemd has its own logging service: systemd-journald called the journal. Every piece of data written to stdout, stderr, logged by the kernel through printk() or by syslog(), all this is sent to the Journal. In addition the journal is very rich with logged information, it doesn’t only log errors returned by a particular service but also related details from your system helping you debug and analyze the reason behind these errors.

Plus, the journal provides you with structured information, making it easily portable. How many times you need to do some coding in order to parse your logs ?!!!! You can use journalctl command for example, with json option in order to have json formatted logs:

sysjson

Cool isn’t it ?

Well this is briefly about systemd, you can find a lot of thing online, articles attacking systemd, others fans of it. I, personnally, like it, except from that  thing of rebooting the whole server after applying new configuration!!!

Try it, there is nothing more refreshing for the mind than exploring new worlds 🙂

Advertisements

Hands on SELinux

Selinux was such a burden for me years ago ( and still sometimes 🙂 ). First time I worked as a sys admin, I had to deploy an apache server with its document Root mounted as an NFS partition. I spent 2 days wondering and searching why in hell my apache couldn’t access its html directory. Too stupid when you found out that there is something called SELinux that is turned on by default to restrict access to your NFS partition by Apache processes!!!!

SELinux is such a wonderful tool for hardening your system and that can get your system to the chaos if you mess with it xD xD xD.

What is SELinux?

SELinux is a built-in kernel module with a bunch of rules and decisions determining who/what has the right to access what. This is done through a labeling of the whole file system, security decisions are then results of a combination of these labels and policy rules used.

It handles access to files/objects much more meticulously than do other classic permissions, ACLs …. But read/write, owner permissions and ACL applies before SELinux rules, meaning that SELinux rules applies only when DAC permissions allow access to an object.

What do you need to know about SELinux?

Suppose you have your kernel compiled with SELinux, and your system is installed with SELinux support, here are some basics to help you get familiar with SELinux (I’m using Centos 6).

Study Case: Apache and SELinux

Now that you can check and see SELinux contexts, let’s understand these contexts with a study case. We’ll go with an apache case since apache is a commonly used server.

First, you need to know that SELinux targeted policy has many rules deciding which process could read/access to file or bind to a port. Apache has its own context, ports, files do too.

The goal of this article is to help get your hand on SELinux and it’s not a recommanded recipe to deploy your web server with SELinux, I’ll write about it later :p

  • Bind to an uncommon port:

Suppose you need Apache to listen on some unusual port than 80 or 8008, you go to your configuration file and alter it with your new port number on the Listen directive(ex:9000), then your restart your server.

Here is the surprise, you can’t start your Apache or force binding to your new port:

bindport

 

Depending on your Linux distru, check your logs, look for lines in audit.log containing type=AVC and string denied:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log

auditlogapa

 

Using semanage command, you can check ports apache could listen to, you won’t find 9000:

semhttpport

 

You can see that the type http_*_t type context isn’t allowed to listen on ports other than 80,443,8080,…

 

No process (type) can access a port/file or other objects if there is not a SELinux rule allowing it.

 

So you need either to give apache the right to access 9000 port by adding this port to ports used by http_port_t type:

#semanage    port    -a    -t   http_port_t   -p   tcp    9000

or simply use a port listed with “semanage port   -l | grep http” , and I prefer the last one.

 

  • Use a DocRoot other than the default /var/www:

 

Well, I’ll go on with port 80. And now I would like to create a vhost for my selinux tutorial and I would like to have my DocumentRoot hosted on the home directory of setuto user (just an example).

I configure my vhost as follows:

vhost

 

Now I reload my apache, surprise: my apache couldn’t see the documentRoot for my newly created vhost:

httprestart

 

Checking the audit.log, you can easily see that the denial is due to SELinux restrictions:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log

auditloghomedir

 

To be sure, use the audit2allow command to check SELinux denials. Now I can find two denials, the first when we used port 9000 and the second when we gave apache a DocumentRoot it’s not allowed to access:

[root@sar ~]# audit2allow -l -a

audit2allow

 

Let’s see why my Apache couldn’t see the directory /home/setuto/www although it exists:

Apache uses the bin file /usr/sbin/httpd , use the following command to check which bin is used by your process:

[root@sar conf.d]# ps faux| grep [h]ttpd

pshttp

Let’s see httpd and my vhost home SELinux context:

[root@sar conf.d]# ls   -lZ  /usr/sbin/httpd

lszhttp

[root@sar conf.d]# ls -lZ /home/setuto/www/

setutowww

You can see from the outputs that SELinux type (or domain) for httpd is httpd_exec_t and for the vhost home directory : user_home_t.

Clearly httpd_exec_t can’t access user_home_t.

 

There are many things you can do to manage this issue:

 

  1. You can change the SELinux type of /usr/sbin/httpd to a type which has the right to access anything: unconfined_*_t, by using:

#chcon    –t     unconfined_exec_t    /usr/sbin/httpd

And it’s too dangerous, since unconfined types could access anything, once your apache is compromised, malicious users can use it to do whatever they want with your system.

 

  1. Or you can simply change the SELinux type of your vhost home to be httpd_sys_content_t same as the type of default DocumentRoot used by your apache which is commonly /var/www/ :

[root@sar conf.d]# ls    -lZ   /var/www/

lszvarwww

By using following commands:

[root@sar conf.d]# chcon  -t httpd_sys_content_t   /home/setuto/www

Check then:

lszsetutoafterchcon

Updates of SELinux context done using chcon are not persistent see “Tips”

By doing this there may be some restrictions forbidden some access to /home/setuto/www since its type is httpd_sys_content_t, think of other processes using this directory. It’s not really recommanded

 

  1. Or may be you can follow audit2allow advices, let’s see them again:

[root@sar ~]# audit2allow -l -a

advisegot

 

So you can either add a rule allowing httpd_t type to access user_home_dir_t . The rule will state the line allow httpd_t user_home_dir_t:dir { search getattr }; (see how to create customized SELinux policy modules)

Or simply activate the rule/Boolean httpd_enable_homedirs allowing apache to access home directories, using the command:

setsebool -P httpd_enable_homedirs on      (-P switch to make persistent changes)

 

To check httpd_enable_homedirs boolean status use getsebool as follows:

getsebool -a | grep httpd_enable_homedir

getsebool

To see what this rule/Boolean is for, use command semanage as follows:

[root@sar setuto]# semanage boolean -l | grep httpd_enable_homedir

semboolean

 

As you can see there is not a unique solution, everything depends on your own policy :p, although I personally prefer audit2allow advices 🙂

I hope this article helps you to see clearly what SELinux are for and how you can manage issues resulting from a bad SELinux configuration by spotting the root cause.

Whenever you can’t run something and you’re sure about your configuration, check SELinux denials, and use semange command to view detailed information and resolve your issue.

Cosmic Birth,