Category Archives: linux

Systemd in the heart of Redhat 7 service management:

If you’re a redhat linux user or you’re using a distribution based on Redhat Linux, you probably wonder what’s in Redhat Linux 7 for you.


Well RHEL 7 brought a lot of new features, beginning with kernel version…. for a sys admin, the big news is the init system, RHEL is now using systemd as the default init system.

What is Systemd?

Systemd is an init system developed by the engineer Lenart Poettering.

If you don’t know, init system is simply the process with PID 1 responsible for spawning all other processes and bringing up the user space within your system. It’s the program your boat loader point to so the kernel uses it to start everything user space.


You certainly had heard of init systems like SysVinit and upstart, systemd is much more elegant 😉


How it works?

Systemd uses the concept of “Unit” to manage services, their dependencies and the allocated resources.

Think of everything services, swap files, mount points, run levels, devices… can be categorized under units. So each device, service or mount point may have a configuration file .unit_name  containing details like binary file to run a service, dependency with others services, mount points….

Systemd Units are as follows:

service, socket, device, mount, automount, swap, target, path, timer, snapshot, slice and scope.

You remember /etc/init.d/ scripts? So think of unit file as init.d scripts..

Apache service for instance, will have a httpd.service unit file which could be configured as follows:


Look at the [service] part, it points to executable file to run/reload/stop the service

With this file, you can configure the run level using this service, look at [Install] part and WantedBy keyword, dependency with other services, observe the [Unit] part and the After keyword. So here, requires apache service.

Target units correspond to classic Runlevels, each target file contains details about required services to be launched. You can have,….

In apache case mentioned above, observe that requirs apache service. In this case and generally, you’ll find a directory /etc/systemd/system/multi-user.wants/ containing a symbolic link pointing to apache service unit and generally to all required services.


What I like the most about systemd :

  • Systemd exploits Cgroups

Systemd uses cgroups kernel to track everything about your processes.

Cgroups are a built-in kernel feature which creates an hierarchy for groups of processes. Think of cgroups as containers that child processes couldn’t escape even with a double fork making it very easy to track them. These containers could serve also for resource management (cpu, ram,…) of a particular process..

If you’re a sys admin, remember how many times you couldn’t determine which service/process spawned a strange process? Or you couldn’t kill all your CGI processes even after shutting down your apache? Well with systemd exploiting cgroups, there is no chance for this to happen J

You can find your cgroups in /sys/fs/cgroups


The command systemd-cgtop returns the top cgroups defined in your system, ordered by their CPU,RAM and load:


  •  Systemd’s journal:

Systemd has its own logging service: systemd-journald called the journal. Every piece of data written to stdout, stderr, logged by the kernel through printk() or by syslog(), all this is sent to the Journal. In addition the journal is very rich with logged information, it doesn’t only log errors returned by a particular service but also related details from your system helping you debug and analyze the reason behind these errors.

Plus, the journal provides you with structured information, making it easily portable. How many times you need to do some coding in order to parse your logs ?!!!! You can use journalctl command for example, with json option in order to have json formatted logs:


Cool isn’t it ?

Well this is briefly about systemd, you can find a lot of thing online, articles attacking systemd, others fans of it. I, personnally, like it, except from that  thing of rebooting the whole server after applying new configuration!!!

Try it, there is nothing more refreshing for the mind than exploring new worlds 🙂


SELinux Contexs

SELinux depends on a group of labels to make access decisions. So when you have your system installed with SELinux, the whole file system is labeled with a context and every file/object has a SELinux context described as the 4-uplet:


So every file, directory, stream, port … within your system has its own context and SELinux rules know which user should access which role to access which type.

When using targeted policy, you only need to understand the user and type in the 4-uplet context.

SELinux users are not the same as users you connect with, many linux users may have the same selinux user within their context.

Let’s have our hands on SELinux contexts:

  • Check SELinux context

The Z switch displays SELinux context of your file,port…

For example:

To check selinux context of a file,directory, use:

ls -lZ




To check selinux context of currently running process, for example apache:

Ps –efZ | grep [h]ttpd




Semanage wonderful command:

I use the semanage command to view and manage SELinux contexts. As I’m working with CentOS 6, I wasn’t it already installed:

Check which package provides the semanage command:

[root@sar ~]# yum provides *bin/semanage
updates | 3.4 kB 00:00
updates/primary_db | 3.5 MB 00:30
updates/filelists_db | 2.2 MB 00:05
policycoreutils-python-2.0.83-19.39.el6.i686 : SELinux policy core python
: utilities
Repo : base
Matched from:
Filename : /usr/sbin/semanage


Install it with:

[root@sar ~]# yum –y install policycoreutils-python


Now let’s try some of semanage options:

Discover the mapping between linux users and SELinux users:

# semanage login -l


Context of a port, for example http ports:

# semanage port -l | grep http


To list context of some files, use semanage as follows and grep on the name of your files:

#semanage fcontext -l | grep drupal



  • Change and modify SELinux contexts:


Commands like chcon and semanage allow you to alter SELinux contexts easily.

If you are just testing a new update, use chcon for temporary changes. With t for type, u for user… man chcon will do for the rest 🙂 :

#chcon –t  httpd_user_content_t /home/setuto/www

Updates using chcon are not persistent, if you relabel your file system or you run restorecon, you will lose the new configuration.


If you want to go for persistent changes, use semanage command instead. This command adds your new updates to SELinux configuration files saving them forever.

For example, use semanage fcontext wih appropriate switch like a to add and d to delete in order to change file’s context.

Here we change the type context of the directory /home/setuto/www from user_home_t to httpd_user_content :

#semanage fcontext -a -t httpd_user_content_t /home/setuto/www


Observe that semanage fcontext doesn’t change the context unless you run restorecon.

The restorecon command is very useful and you can use it to restore contexts modified by chcon command, restorecon returns back to last modification saved into selinux configuration files.

To compare current contexts with the default one, use matchpathcon command with filename.



For further details, man command will do 😉


Hands on SELinux

Selinux was such a burden for me years ago ( and still sometimes 🙂 ). First time I worked as a sys admin, I had to deploy an apache server with its document Root mounted as an NFS partition. I spent 2 days wondering and searching why in hell my apache couldn’t access its html directory. Too stupid when you found out that there is something called SELinux that is turned on by default to restrict access to your NFS partition by Apache processes!!!!

SELinux is such a wonderful tool for hardening your system and that can get your system to the chaos if you mess with it xD xD xD.

What is SELinux?

SELinux is a built-in kernel module with a bunch of rules and decisions determining who/what has the right to access what. This is done through a labeling of the whole file system, security decisions are then results of a combination of these labels and policy rules used.

It handles access to files/objects much more meticulously than do other classic permissions, ACLs …. But read/write, owner permissions and ACL applies before SELinux rules, meaning that SELinux rules applies only when DAC permissions allow access to an object.

What do you need to know about SELinux?

Suppose you have your kernel compiled with SELinux, and your system is installed with SELinux support, here are some basics to help you get familiar with SELinux (I’m using Centos 6).

Study Case: Apache and SELinux

Now that you can check and see SELinux contexts, let’s understand these contexts with a study case. We’ll go with an apache case since apache is a commonly used server.

First, you need to know that SELinux targeted policy has many rules deciding which process could read/access to file or bind to a port. Apache has its own context, ports, files do too.

The goal of this article is to help get your hand on SELinux and it’s not a recommanded recipe to deploy your web server with SELinux, I’ll write about it later :p

  • Bind to an uncommon port:

Suppose you need Apache to listen on some unusual port than 80 or 8008, you go to your configuration file and alter it with your new port number on the Listen directive(ex:9000), then your restart your server.

Here is the surprise, you can’t start your Apache or force binding to your new port:



Depending on your Linux distru, check your logs, look for lines in audit.log containing type=AVC and string denied:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log



Using semanage command, you can check ports apache could listen to, you won’t find 9000:



You can see that the type http_*_t type context isn’t allowed to listen on ports other than 80,443,8080,…


No process (type) can access a port/file or other objects if there is not a SELinux rule allowing it.


So you need either to give apache the right to access 9000 port by adding this port to ports used by http_port_t type:

#semanage    port    -a    -t   http_port_t   -p   tcp    9000

or simply use a port listed with “semanage port   -l | grep http” , and I prefer the last one.


  • Use a DocRoot other than the default /var/www:


Well, I’ll go on with port 80. And now I would like to create a vhost for my selinux tutorial and I would like to have my DocumentRoot hosted on the home directory of setuto user (just an example).

I configure my vhost as follows:



Now I reload my apache, surprise: my apache couldn’t see the documentRoot for my newly created vhost:



Checking the audit.log, you can easily see that the denial is due to SELinux restrictions:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log



To be sure, use the audit2allow command to check SELinux denials. Now I can find two denials, the first when we used port 9000 and the second when we gave apache a DocumentRoot it’s not allowed to access:

[root@sar ~]# audit2allow -l -a



Let’s see why my Apache couldn’t see the directory /home/setuto/www although it exists:

Apache uses the bin file /usr/sbin/httpd , use the following command to check which bin is used by your process:

[root@sar conf.d]# ps faux| grep [h]ttpd


Let’s see httpd and my vhost home SELinux context:

[root@sar conf.d]# ls   -lZ  /usr/sbin/httpd


[root@sar conf.d]# ls -lZ /home/setuto/www/


You can see from the outputs that SELinux type (or domain) for httpd is httpd_exec_t and for the vhost home directory : user_home_t.

Clearly httpd_exec_t can’t access user_home_t.


There are many things you can do to manage this issue:


  1. You can change the SELinux type of /usr/sbin/httpd to a type which has the right to access anything: unconfined_*_t, by using:

#chcon    –t     unconfined_exec_t    /usr/sbin/httpd

And it’s too dangerous, since unconfined types could access anything, once your apache is compromised, malicious users can use it to do whatever they want with your system.


  1. Or you can simply change the SELinux type of your vhost home to be httpd_sys_content_t same as the type of default DocumentRoot used by your apache which is commonly /var/www/ :

[root@sar conf.d]# ls    -lZ   /var/www/


By using following commands:

[root@sar conf.d]# chcon  -t httpd_sys_content_t   /home/setuto/www

Check then:


Updates of SELinux context done using chcon are not persistent see “Tips”

By doing this there may be some restrictions forbidden some access to /home/setuto/www since its type is httpd_sys_content_t, think of other processes using this directory. It’s not really recommanded


  1. Or may be you can follow audit2allow advices, let’s see them again:

[root@sar ~]# audit2allow -l -a



So you can either add a rule allowing httpd_t type to access user_home_dir_t . The rule will state the line allow httpd_t user_home_dir_t:dir { search getattr }; (see how to create customized SELinux policy modules)

Or simply activate the rule/Boolean httpd_enable_homedirs allowing apache to access home directories, using the command:

setsebool -P httpd_enable_homedirs on      (-P switch to make persistent changes)


To check httpd_enable_homedirs boolean status use getsebool as follows:

getsebool -a | grep httpd_enable_homedir


To see what this rule/Boolean is for, use command semanage as follows:

[root@sar setuto]# semanage boolean -l | grep httpd_enable_homedir



As you can see there is not a unique solution, everything depends on your own policy :p, although I personally prefer audit2allow advices 🙂

I hope this article helps you to see clearly what SELinux are for and how you can manage issues resulting from a bad SELinux configuration by spotting the root cause.

Whenever you can’t run something and you’re sure about your configuration, check SELinux denials, and use semange command to view detailed information and resolve your issue.

Cosmic Birth,


SELinux Configuration

With SELinux activated on your kernel, you need to know which mode and policy (ex: targeted, MLS) you are running.

SELinux mode is used to enable/disable SELinux, whereas the policy is about the philosophy used to enhance security within your system, targeted is the commonly used policy and also the default one.

SELinux Modes are:

  • Enforcing: respect your policy rules.
  • Permissive:This  mode will not use policy rules to deny access but it will log everything the policy would prevent.
  • Disabled.

When use Permissive mode?

If your sysem wasn’t initially using SELinux and you would like to go on with Enforcing mode, use permissive mode and audit all the denials you get when running your applications.

If auditd is installed and running, check audit.log logs for type=AVC messages with denied string:

#grep denied /var/log/audit/audit.log



You can also use audit2allow command to check SELinux denials and follow the output recommandations:




You can then go on with Enforcing mode after some work on your policy adding and updating the rules you need to correctly run your applications.

You should reboot your OS if you go from permissive to disabled mode and vice-versa.


Here is your set of tools to manipulate SELinux mode and policy:


  • sestatus command:

Use sestatus command to see the current configuration in your system.
you can see below I’m using enforcing mode and targeted policy.




  • getenforce, setenforce commands

getenforce command returns SELinux mode currently running, and you can configure the mode with setenforce {0,1} command as follows:




You can also configure your SELinux mode and policy by editing the appropriate section in the /etc/selinux/config file:


# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing – SELinux security policy is enforced.

#     permissive – SELinux prints warnings instead of enforcing.

#     disabled – No SELinux policy is loaded.


# SELINUXTYPE= can take one of these two values:

#     targeted – Targeted processes are protected,

#     mls – Multi Level Security protection.



Don’t freak out if SELinux policy lacks some necessary rules for your application :), you can always add new rules by creating SELinux modules using audit2allow , but this is not the goal of this post. I’ll write another one for customized policies :p




ACL: Need particular permissions for a user not in owner/groupe of a file?

Have you ever needed to grant some users more permissions on some files, but the user doesn’t belong to the owner group?  First, you feel tempted to increase “Others” permission or may be add your user to the owner group: not really what you want? not so secure? Try ACLs :).

ACLs allow you to grant permissions you need to users not belonging to the file’s owner and group without altering Others permissions.

To use ACLs on some files/directories, the partition containg these files should be mounted with acl enabled.

Let’s say, I need to give “admin” user read/write permissions on the file /redologs/mydb/u01/control01.ctl.

as ACLs are not enabled on /redologs, look at my fstab:

cat /etc/fstab | grep redologs

/dev/vgdata/lvredo                /redologs                               ext4        defaults                   1    1


First, we need to enable ACL on  /redo_logs by remounting it with acl option. Here we do it on runtime:

mount -o remount,acl    /dev/mapper/vgdate-lvredo      /redologs

Let’s check current permissions before altering the file permissions:

cd /redologs/mydb/u01/

getfacl control01.ctl

#file: control01.ctl

#owner: oracle

#group: oinstall





Now, let’s grant our user read/write permission on control01.ctl.

setfacl -m -u:admin:rw   control01.ctl

#file: control01.ctl

#owner: oracle

#group: oinstall






If you do a ls -l to the file, you will notice the + symbol:

ls -l control01.ctl
-rw-rw----+ oracle oinstall 3198537 Apr 10 14:25 control01.ctl

Connect with admin user and test 🙂

Now, here a brief how-to play with ACLs:

– use getfacl filename to diplay current acl on the file.
– To update or set new ACL on a file, use setfacl as following:

setfacl -m rules filename

where rules are:

u:uid/usernme:permissions(rwx) for a user
g:gid/groupname:permissions(rwx) for a group
o:permissions(rwx) fro other

– To remove all permissions, use -x option:

setfacl -x filename

If you need to remove specific permission, use -x and specify the rule:

setfacl -x rules filename

Well it seems quick :), first time I used ACLs, I as working with Gentoo distribution and I had to recompile the whole kernel with ACL support to be able to use it on my filesystems and restart the server. Not so tempting to your client, no? xD xD