Author Archives: Cosmic Birth

I acknowledge that sysadmin is a fascinating job, and as an engineer, it pulls the best of me because I always have to look at thing I never knew or understood. This blog is to report things I did or discovered although it’s pretty difficult to report everything.

I’m also passionate about Linux, how it’s transparent and easy to approach things in it and most importantly it alows you to do things the way you like.

So this is a pseudo diary…

Systemd in the heart of Redhat 7 service management:

If you’re a redhat linux user or you’re using a distribution based on Redhat Linux, you probably wonder what’s in Redhat Linux 7 for you.

7cent

Well RHEL 7 brought a lot of new features, beginning with kernel version…. for a sys admin, the big news is the init system, RHEL is now using systemd as the default init system.

What is Systemd?

Systemd is an init system developed by the engineer Lenart Poettering.

If you don’t know, init system is simply the process with PID 1 responsible for spawning all other processes and bringing up the user space within your system. It’s the program your boat loader point to so the kernel uses it to start everything user space.

btprc

You certainly had heard of init systems like SysVinit and upstart, systemd is much more elegant 😉

 

How it works?

Systemd uses the concept of “Unit” to manage services, their dependencies and the allocated resources.

Think of everything services, swap files, mount points, run levels, devices… can be categorized under units. So each device, service or mount point may have a configuration file .unit_name  containing details like binary file to run a service, dependency with others services, mount points….

Systemd Units are as follows:

service, socket, device, mount, automount, swap, target, path, timer, snapshot, slice and scope.

You remember /etc/init.d/ scripts? So think of unit file as init.d scripts..

Apache service for instance, will have a httpd.service unit file which could be configured as follows:

httunit

Look at the [service] part, it points to executable file to run/reload/stop the service

With this file, you can configure the run level using this service, look at [Install] part and WantedBy keyword, dependency with other services, observe the [Unit] part and the After keyword. So here, multi-user.target requires apache service.

Target units correspond to classic Runlevels, each target file contains details about required services to be launched. You can have multi-user.target, graphical.target….

In apache case mentioned above, observe that multi-user.target requirs apache service. In this case and generally, you’ll find a directory /etc/systemd/system/multi-user.wants/ containing a symbolic link pointing to apache service unit and generally to all required services.

 

What I like the most about systemd :

  • Systemd exploits Cgroups

Systemd uses cgroups kernel to track everything about your processes.

Cgroups are a built-in kernel feature which creates an hierarchy for groups of processes. Think of cgroups as containers that child processes couldn’t escape even with a double fork making it very easy to track them. These containers could serve also for resource management (cpu, ram,…) of a particular process..

If you’re a sys admin, remember how many times you couldn’t determine which service/process spawned a strange process? Or you couldn’t kill all your CGI processes even after shutting down your apache? Well with systemd exploiting cgroups, there is no chance for this to happen J

You can find your cgroups in /sys/fs/cgroups

cgroup

The command systemd-cgtop returns the top cgroups defined in your system, ordered by their CPU,RAM and load:

systemd-cgtop

  •  Systemd’s journal:

Systemd has its own logging service: systemd-journald called the journal. Every piece of data written to stdout, stderr, logged by the kernel through printk() or by syslog(), all this is sent to the Journal. In addition the journal is very rich with logged information, it doesn’t only log errors returned by a particular service but also related details from your system helping you debug and analyze the reason behind these errors.

Plus, the journal provides you with structured information, making it easily portable. How many times you need to do some coding in order to parse your logs ?!!!! You can use journalctl command for example, with json option in order to have json formatted logs:

sysjson

Cool isn’t it ?

Well this is briefly about systemd, you can find a lot of thing online, articles attacking systemd, others fans of it. I, personnally, like it, except from that  thing of rebooting the whole server after applying new configuration!!!

Try it, there is nothing more refreshing for the mind than exploring new worlds 🙂

Advertisements

SELinux Contexs

SELinux depends on a group of labels to make access decisions. So when you have your system installed with SELinux, the whole file system is labeled with a context and every file/object has a SELinux context described as the 4-uplet:

(user,role,type,range)

So every file, directory, stream, port … within your system has its own context and SELinux rules know which user should access which role to access which type.

When using targeted policy, you only need to understand the user and type in the 4-uplet context.

SELinux users are not the same as users you connect with, many linux users may have the same selinux user within their context.

Let’s have our hands on SELinux contexts:

  • Check SELinux context

The Z switch displays SELinux context of your file,port…

For example:

To check selinux context of a file,directory, use:

ls -lZ

 

lslz

 

To check selinux context of currently running process, for example apache:

Ps –efZ | grep [h]ttpd

 

psezf

 

Semanage wonderful command:

I use the semanage command to view and manage SELinux contexts. As I’m working with CentOS 6, I wasn’t it already installed:

Check which package provides the semanage command:

[root@sar ~]# yum provides *bin/semanage
updates | 3.4 kB 00:00
updates/primary_db | 3.5 MB 00:30
updates/filelists_db | 2.2 MB 00:05
policycoreutils-python-2.0.83-19.39.el6.i686 : SELinux policy core python
: utilities
Repo : base
Matched from:
Filename : /usr/sbin/semanage

 

Install it with:

[root@sar ~]# yum –y install policycoreutils-python

 

Now let’s try some of semanage options:

Discover the mapping between linux users and SELinux users:

# semanage login -l

semanagelogin

Context of a port, for example http ports:

# semanage port -l | grep http

semport

To list context of some files, use semanage as follows and grep on the name of your files:

#semanage fcontext -l | grep drupal

semfcont

 

  • Change and modify SELinux contexts:

 

Commands like chcon and semanage allow you to alter SELinux contexts easily.

If you are just testing a new update, use chcon for temporary changes. With t for type, u for user… man chcon will do for the rest 🙂 :

#chcon –t  httpd_user_content_t /home/setuto/www

Updates using chcon are not persistent, if you relabel your file system or you run restorecon, you will lose the new configuration.

 

If you want to go for persistent changes, use semanage command instead. This command adds your new updates to SELinux configuration files saving them forever.

For example, use semanage fcontext wih appropriate switch like a to add and d to delete in order to change file’s context.

Here we change the type context of the directory /home/setuto/www from user_home_t to httpd_user_content :

#semanage fcontext -a -t httpd_user_content_t /home/setuto/www

semfcon-restre

Observe that semanage fcontext doesn’t change the context unless you run restorecon.

The restorecon command is very useful and you can use it to restore contexts modified by chcon command, restorecon returns back to last modification saved into selinux configuration files.

To compare current contexts with the default one, use matchpathcon command with filename.

Example:

matchpathcon

For further details, man command will do 😉

 

Hands on SELinux

Selinux was such a burden for me years ago ( and still sometimes 🙂 ). First time I worked as a sys admin, I had to deploy an apache server with its document Root mounted as an NFS partition. I spent 2 days wondering and searching why in hell my apache couldn’t access its html directory. Too stupid when you found out that there is something called SELinux that is turned on by default to restrict access to your NFS partition by Apache processes!!!!

SELinux is such a wonderful tool for hardening your system and that can get your system to the chaos if you mess with it xD xD xD.

What is SELinux?

SELinux is a built-in kernel module with a bunch of rules and decisions determining who/what has the right to access what. This is done through a labeling of the whole file system, security decisions are then results of a combination of these labels and policy rules used.

It handles access to files/objects much more meticulously than do other classic permissions, ACLs …. But read/write, owner permissions and ACL applies before SELinux rules, meaning that SELinux rules applies only when DAC permissions allow access to an object.

What do you need to know about SELinux?

Suppose you have your kernel compiled with SELinux, and your system is installed with SELinux support, here are some basics to help you get familiar with SELinux (I’m using Centos 6).

Study Case: Apache and SELinux

Now that you can check and see SELinux contexts, let’s understand these contexts with a study case. We’ll go with an apache case since apache is a commonly used server.

First, you need to know that SELinux targeted policy has many rules deciding which process could read/access to file or bind to a port. Apache has its own context, ports, files do too.

The goal of this article is to help get your hand on SELinux and it’s not a recommanded recipe to deploy your web server with SELinux, I’ll write about it later :p

  • Bind to an uncommon port:

Suppose you need Apache to listen on some unusual port than 80 or 8008, you go to your configuration file and alter it with your new port number on the Listen directive(ex:9000), then your restart your server.

Here is the surprise, you can’t start your Apache or force binding to your new port:

bindport

 

Depending on your Linux distru, check your logs, look for lines in audit.log containing type=AVC and string denied:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log

auditlogapa

 

Using semanage command, you can check ports apache could listen to, you won’t find 9000:

semhttpport

 

You can see that the type http_*_t type context isn’t allowed to listen on ports other than 80,443,8080,…

 

No process (type) can access a port/file or other objects if there is not a SELinux rule allowing it.

 

So you need either to give apache the right to access 9000 port by adding this port to ports used by http_port_t type:

#semanage    port    -a    -t   http_port_t   -p   tcp    9000

or simply use a port listed with “semanage port   -l | grep http” , and I prefer the last one.

 

  • Use a DocRoot other than the default /var/www:

 

Well, I’ll go on with port 80. And now I would like to create a vhost for my selinux tutorial and I would like to have my DocumentRoot hosted on the home directory of setuto user (just an example).

I configure my vhost as follows:

vhost

 

Now I reload my apache, surprise: my apache couldn’t see the documentRoot for my newly created vhost:

httprestart

 

Checking the audit.log, you can easily see that the denial is due to SELinux restrictions:

[root@sar ~]# tail -n 2 /var/log/audit/audit.log

auditloghomedir

 

To be sure, use the audit2allow command to check SELinux denials. Now I can find two denials, the first when we used port 9000 and the second when we gave apache a DocumentRoot it’s not allowed to access:

[root@sar ~]# audit2allow -l -a

audit2allow

 

Let’s see why my Apache couldn’t see the directory /home/setuto/www although it exists:

Apache uses the bin file /usr/sbin/httpd , use the following command to check which bin is used by your process:

[root@sar conf.d]# ps faux| grep [h]ttpd

pshttp

Let’s see httpd and my vhost home SELinux context:

[root@sar conf.d]# ls   -lZ  /usr/sbin/httpd

lszhttp

[root@sar conf.d]# ls -lZ /home/setuto/www/

setutowww

You can see from the outputs that SELinux type (or domain) for httpd is httpd_exec_t and for the vhost home directory : user_home_t.

Clearly httpd_exec_t can’t access user_home_t.

 

There are many things you can do to manage this issue:

 

  1. You can change the SELinux type of /usr/sbin/httpd to a type which has the right to access anything: unconfined_*_t, by using:

#chcon    –t     unconfined_exec_t    /usr/sbin/httpd

And it’s too dangerous, since unconfined types could access anything, once your apache is compromised, malicious users can use it to do whatever they want with your system.

 

  1. Or you can simply change the SELinux type of your vhost home to be httpd_sys_content_t same as the type of default DocumentRoot used by your apache which is commonly /var/www/ :

[root@sar conf.d]# ls    -lZ   /var/www/

lszvarwww

By using following commands:

[root@sar conf.d]# chcon  -t httpd_sys_content_t   /home/setuto/www

Check then:

lszsetutoafterchcon

Updates of SELinux context done using chcon are not persistent see “Tips”

By doing this there may be some restrictions forbidden some access to /home/setuto/www since its type is httpd_sys_content_t, think of other processes using this directory. It’s not really recommanded

 

  1. Or may be you can follow audit2allow advices, let’s see them again:

[root@sar ~]# audit2allow -l -a

advisegot

 

So you can either add a rule allowing httpd_t type to access user_home_dir_t . The rule will state the line allow httpd_t user_home_dir_t:dir { search getattr }; (see how to create customized SELinux policy modules)

Or simply activate the rule/Boolean httpd_enable_homedirs allowing apache to access home directories, using the command:

setsebool -P httpd_enable_homedirs on      (-P switch to make persistent changes)

 

To check httpd_enable_homedirs boolean status use getsebool as follows:

getsebool -a | grep httpd_enable_homedir

getsebool

To see what this rule/Boolean is for, use command semanage as follows:

[root@sar setuto]# semanage boolean -l | grep httpd_enable_homedir

semboolean

 

As you can see there is not a unique solution, everything depends on your own policy :p, although I personally prefer audit2allow advices 🙂

I hope this article helps you to see clearly what SELinux are for and how you can manage issues resulting from a bad SELinux configuration by spotting the root cause.

Whenever you can’t run something and you’re sure about your configuration, check SELinux denials, and use semange command to view detailed information and resolve your issue.

Cosmic Birth,

 

SELinux Configuration

With SELinux activated on your kernel, you need to know which mode and policy (ex: targeted, MLS) you are running.

SELinux mode is used to enable/disable SELinux, whereas the policy is about the philosophy used to enhance security within your system, targeted is the commonly used policy and also the default one.

SELinux Modes are:

  • Enforcing: respect your policy rules.
  • Permissive:This  mode will not use policy rules to deny access but it will log everything the policy would prevent.
  • Disabled.

When use Permissive mode?

If your sysem wasn’t initially using SELinux and you would like to go on with Enforcing mode, use permissive mode and audit all the denials you get when running your applications.

If auditd is installed and running, check audit.log logs for type=AVC messages with denied string:

#grep denied /var/log/audit/audit.log

auditlog

 

You can also use audit2allow command to check SELinux denials and follow the output recommandations:

 

audit2allow

 

You can then go on with Enforcing mode after some work on your policy adding and updating the rules you need to correctly run your applications.

You should reboot your OS if you go from permissive to disabled mode and vice-versa.

 

Here is your set of tools to manipulate SELinux mode and policy:

 

  • sestatus command:

Use sestatus command to see the current configuration in your system.
you can see below I’m using enforcing mode and targeted policy.

 

sesatus

 

  • getenforce, setenforce commands

getenforce command returns SELinux mode currently running, and you can configure the mode with setenforce {0,1} command as follows:

 

getsetenforce

 

You can also configure your SELinux mode and policy by editing the appropriate section in the /etc/selinux/config file:

 

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

#     enforcing – SELinux security policy is enforced.

#     permissive – SELinux prints warnings instead of enforcing.

#     disabled – No SELinux policy is loaded.

SELINUX=enforcing

# SELINUXTYPE= can take one of these two values:

#     targeted – Targeted processes are protected,

#     mls – Multi Level Security protection.

SELINUXTYPE=targeted

 

Don’t freak out if SELinux policy lacks some necessary rules for your application :), you can always add new rules by creating SELinux modules using audit2allow , but this is not the goal of this post. I’ll write another one for customized policies :p

 

 

 

ACL: Need particular permissions for a user not in owner/groupe of a file?

Have you ever needed to grant some users more permissions on some files, but the user doesn’t belong to the owner group?  First, you feel tempted to increase “Others” permission or may be add your user to the owner group: not really what you want? not so secure? Try ACLs :).

ACLs allow you to grant permissions you need to users not belonging to the file’s owner and group without altering Others permissions.

To use ACLs on some files/directories, the partition containg these files should be mounted with acl enabled.

Let’s say, I need to give “admin” user read/write permissions on the file /redologs/mydb/u01/control01.ctl.

as ACLs are not enabled on /redologs, look at my fstab:

cat /etc/fstab | grep redologs

/dev/vgdata/lvredo                /redologs                               ext4        defaults                   1    1

 

First, we need to enable ACL on  /redo_logs by remounting it with acl option. Here we do it on runtime:

mount -o remount,acl    /dev/mapper/vgdate-lvredo      /redologs

Let’s check current permissions before altering the file permissions:

cd /redologs/mydb/u01/

getfacl control01.ctl

#file: control01.ctl

#owner: oracle

#group: oinstall

user::rw-

group::r--

other::---

 

Now, let’s grant our user read/write permission on control01.ctl.

setfacl -m -u:admin:rw   control01.ctl

#file: control01.ctl

#owner: oracle

#group: oinstall

user::rw-

user:admin:rw-

group::r--

mask::rw-

other::---

If you do a ls -l to the file, you will notice the + symbol:

ls -l control01.ctl
-rw-rw----+ oracle oinstall 3198537 Apr 10 14:25 control01.ctl

Connect with admin user and test 🙂

Now, here a brief how-to play with ACLs:

– use getfacl filename to diplay current acl on the file.
– To update or set new ACL on a file, use setfacl as following:

setfacl -m rules filename

where rules are:

u:uid/usernme:permissions(rwx) for a user
g:gid/groupname:permissions(rwx) for a group
o:permissions(rwx) fro other

– To remove all permissions, use -x option:

setfacl -x filename

If you need to remove specific permission, use -x and specify the rule:

setfacl -x rules filename

Well it seems quick :), first time I used ACLs, I as working with Gentoo distribution and I had to recompile the whole kernel with ACL support to be able to use it on my filesystems and restart the server. Not so tempting to your client, no? xD xD

Firebug

I’m a Mozilla Firefox user and I love it. One of the most awesome adds is Firebug. You can use it to debug your long time taking website or debug your JavaScript codes.

As for me, I used with clients. I get a lot of clients calls asking me why their sites are taking too much time to load. Almost all the time, the web server is well performing and the machine load is normal. But still the client won’t understand unless you show them “evidences”. And that’s what Firebug are for :D. It saves my life when at phone the client won’t believe in you LOL.

I run the site and Firebug at the same time:

Fireb

Then I choose the net tab, all and look for the HTTP request/object taking too long to load. I point the cursor on and get the beautiful graphics breaking the time the query took into pieces of:

1- Horizontal and colorful graphics:

graphs

  • Blocking: The time that your environment takes to open a TCP connection
  • DNS lookup: The time name resolution takes,
  • Connecting: The time needed to establish a connection with the server. Here, you can notice the difference between having a keep alive connection and none.
  • Sending: Time needed to send the request to the web server,
  • Waiting:Time needed to receive the first byte/answer from the web server,
  • Receiving: Time needed to download the response/object from the server.

If the web server is taking too long to process your request then you should be able to spot it, just notice the “waiting time” value.

2-Vertical time lines:

line

The DomContentLoad and the load time. If you have some JavaScript code running on your website, these lines help you understand how JS events are fired.

This post: “Page load analysis” is a very good and detailed about Firebug and it goes with code simulation of many cases,

DRBD Performance (draft)

DRBD can be used as RAID 1 but still couldn’t perform as well. More than RAID1, its performance rely strongly on hardware performance.

Let’s break DRBD replication into steps:

1. Data transfer through network:

Plus having a good bandwidth, DRBD documentation suggest some parameter to fine tune the net performance such as:

– The rate of synchronization,

– Using checksum-based synchronization,

– Replication modes,

2. Read/write data on the disk:

When I deployed DRBD, I had 2*8To SSD and gigabyte ethernet card and still not statisfied. To tackle I/O latency, you should consider at least two parameter:

– The I/O scheduler as it comes between DRBD binary and the disk:

The official docs suggest using the deadline I/O scheduler, although I think DRBD replication(RAID1 like) uses extensive write operations and deadline does prefer read operations. For me, I’ve chosen no I/O scheduler because I used SSD disks and kernel I/O schedulers are here for HDD only xd.

– The read/write speed of the disk:

I used Bonnie++ in order to do a benchmark of the disks I may use, SSD, although are expensive,  are very good.

Replicate your data using DRBD (draft)

Some programs just don’t include replication as an option. DRBD is then a very good way to replicate your service/data transparently.

Just have your service and configuration files/data run on a disk, and deploy DRBD to replicate the disk on another server. You can use heartbeat or corosync along to assure high availability to your system.

What’s amazing is that DRBD is OPEN SOURCE.

To deploy DRBD, on both nodes simply follow:

– ReCompile and build your kernel with DRBD support:

I’ve tested DRBD on Gentoo distribution,

# cd /usr/src/linux-3.10.7-gentoo/
linux-3.10.7-gentoo # make menuconfig

scripts/kconfig/mconf Kconfig
#
# using defaults found in /boot/config-3.4.45-sdf134-core2-64
#

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

DRBD
Compile and build your binary kernel, then copy it and link it to where your boot-loader is configured to look
for the kernel.

 # cat /proc/cpuinfo
linux-3.10.7-gentoo # time make -j4
linux-3.10.7-gentoo # make modules_install
linux-3.10.7-gentoo # cp arch/x86/boot/linux-3.10.7 /boot/
#reboot

– Synhronize time with an NTP server:

</pre>
emerge -tav <a class="external text" href="http://packages.gentoo.org/package/net-misc/ntp" rel="nofollow">net-misc/ntp</a>

and follow Gentoo wiki.

– Configure network connectivity on both nodes.

I suggest using dedicated network interface for DRBD. Then use hosts file so both nodes could connect  to and identify each other.

# echo '192.168.1.10 node1' >> /etc/hosts

Chek if no service is using port 7788 and 7799, no firewall rule is blocking in/out tcp connection
between nodes.

– Install DRBD control tools

</pre>
# emerge -tav sys-cluster/drbd

* IMPORTANT: 8 news items need reading for repository 'gentoo'.
* Use eselect news to read news items.

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild  N     ] sys-cluster/drbd-8.4.2  USE="udev -bash-completion -heartbeat -pacemaker -xen" 660 kB

Total: 1 package (1 new), Size of downloads: 660 kB

Would you like to merge these packages? [Yes/No]

– Prepare your disks:

Partition your disk  and hand it empty to DRBD without a filesystem.

</pre>
# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.21.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help):

– Configure your resources:

DRBD is configured through /etc/drbd.d/global_common.conf and use /etc/drbd.conf.

The resource is configured through the file /etc/drbd.d/res_name.res.

Refer to DRBD Doc for more details.

– Start DRBD and create metadata on your disk

</pre>
# /etc/init.d/drbd start

#drbdadm create-md res_name0

#drbdadm primary res_name0  #Only on the primary node

You can then write on your DRBD device, format it and mount it, but you can do it only on the primary node.

– Watch DRBD resource synhronization using the proc file:

# cat /proc/drbd*

Troubleshooting?

It’s simple and straightforward, there is still one more thing you should consider: performance.

Monitoring DRBD status (draft)

You surely thought should I always check DRBD status by reading the bunch of line provided by:

cat /proc/drbd

DRBD monitoring is a must doing so you can ensure you’re data is consistent and uptodate. I worked with DRBD 8.4 and tried to figure out something that check the whole DRBD status.

I mean the folloing by conserving the order:

  1. Split brain
  2. Connection status
  3. Ressource Role
  4. Disk status
  5. I/O status
  6. Performance

I’m no DRBD expert, I just tried think something logical to my monitoring would be efficient and include no redundant alarm. And yes, we, mathematicians, adore optimization 🙂 :).

The flow of checks goes like in the following chart:

monitor drdbd logic