Monthly Archives: September 2013

Firebug

I’m a Mozilla Firefox user and I love it. One of the most awesome adds is Firebug. You can use it to debug your long time taking website or debug your JavaScript codes.

As for me, I used with clients. I get a lot of clients calls asking me why their sites are taking too much time to load. Almost all the time, the web server is well performing and the machine load is normal. But still the client won’t understand unless you show them “evidences”. And that’s what Firebug are for :D. It saves my life when at phone the client won’t believe in you LOL.

I run the site and Firebug at the same time:

Fireb

Then I choose the net tab, all and look for the HTTP request/object taking too long to load. I point the cursor on and get the beautiful graphics breaking the time the query took into pieces of:

1- Horizontal and colorful graphics:

graphs

  • Blocking: The time that your environment takes to open a TCP connection
  • DNS lookup: The time name resolution takes,
  • Connecting: The time needed to establish a connection with the server. Here, you can notice the difference between having a keep alive connection and none.
  • Sending: Time needed to send the request to the web server,
  • Waiting:Time needed to receive the first byte/answer from the web server,
  • Receiving: Time needed to download the response/object from the server.

If the web server is taking too long to process your request then you should be able to spot it, just notice the “waiting time” value.

2-Vertical time lines:

line

The DomContentLoad and the load time. If you have some JavaScript code running on your website, these lines help you understand how JS events are fired.

This post: “Page load analysis” is a very good and detailed about Firebug and it goes with code simulation of many cases,

DRBD Performance (draft)

DRBD can be used as RAID 1 but still couldn’t perform as well. More than RAID1, its performance rely strongly on hardware performance.

Let’s break DRBD replication into steps:

1. Data transfer through network:

Plus having a good bandwidth, DRBD documentation suggest some parameter to fine tune the net performance such as:

– The rate of synchronization,

– Using checksum-based synchronization,

– Replication modes,

2. Read/write data on the disk:

When I deployed DRBD, I had 2*8To SSD and gigabyte ethernet card and still not statisfied. To tackle I/O latency, you should consider at least two parameter:

– The I/O scheduler as it comes between DRBD binary and the disk:

The official docs suggest using the deadline I/O scheduler, although I think DRBD replication(RAID1 like) uses extensive write operations and deadline does prefer read operations. For me, I’ve chosen no I/O scheduler because I used SSD disks and kernel I/O schedulers are here for HDD only xd.

– The read/write speed of the disk:

I used Bonnie++ in order to do a benchmark of the disks I may use, SSD, although are expensive,  are very good.

Replicate your data using DRBD (draft)

Some programs just don’t include replication as an option. DRBD is then a very good way to replicate your service/data transparently.

Just have your service and configuration files/data run on a disk, and deploy DRBD to replicate the disk on another server. You can use heartbeat or corosync along to assure high availability to your system.

What’s amazing is that DRBD is OPEN SOURCE.

To deploy DRBD, on both nodes simply follow:

– ReCompile and build your kernel with DRBD support:

I’ve tested DRBD on Gentoo distribution,

# cd /usr/src/linux-3.10.7-gentoo/
linux-3.10.7-gentoo # make menuconfig

scripts/kconfig/mconf Kconfig
#
# using defaults found in /boot/config-3.4.45-sdf134-core2-64
#

*** End of the configuration.
*** Execute 'make' to start the build or try 'make help'.

DRBD
Compile and build your binary kernel, then copy it and link it to where your boot-loader is configured to look
for the kernel.

 # cat /proc/cpuinfo
linux-3.10.7-gentoo # time make -j4
linux-3.10.7-gentoo # make modules_install
linux-3.10.7-gentoo # cp arch/x86/boot/linux-3.10.7 /boot/
#reboot

– Synhronize time with an NTP server:

</pre>
emerge -tav <a class="external text" href="http://packages.gentoo.org/package/net-misc/ntp" rel="nofollow">net-misc/ntp</a>

and follow Gentoo wiki.

– Configure network connectivity on both nodes.

I suggest using dedicated network interface for DRBD. Then use hosts file so both nodes could connect  to and identify each other.

# echo '192.168.1.10 node1' >> /etc/hosts

Chek if no service is using port 7788 and 7799, no firewall rule is blocking in/out tcp connection
between nodes.

– Install DRBD control tools

</pre>
# emerge -tav sys-cluster/drbd

* IMPORTANT: 8 news items need reading for repository 'gentoo'.
* Use eselect news to read news items.

These are the packages that would be merged, in reverse order:

Calculating dependencies... done!
[ebuild  N     ] sys-cluster/drbd-8.4.2  USE="udev -bash-completion -heartbeat -pacemaker -xen" 660 kB

Total: 1 package (1 new), Size of downloads: 660 kB

Would you like to merge these packages? [Yes/No]

– Prepare your disks:

Partition your disk  and hand it empty to DRBD without a filesystem.

</pre>
# fdisk /dev/sdb

Welcome to fdisk (util-linux 2.21.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help):

– Configure your resources:

DRBD is configured through /etc/drbd.d/global_common.conf and use /etc/drbd.conf.

The resource is configured through the file /etc/drbd.d/res_name.res.

Refer to DRBD Doc for more details.

– Start DRBD and create metadata on your disk

</pre>
# /etc/init.d/drbd start

#drbdadm create-md res_name0

#drbdadm primary res_name0  #Only on the primary node

You can then write on your DRBD device, format it and mount it, but you can do it only on the primary node.

– Watch DRBD resource synhronization using the proc file:

# cat /proc/drbd*

Troubleshooting?

It’s simple and straightforward, there is still one more thing you should consider: performance.

Monitoring DRBD status (draft)

You surely thought should I always check DRBD status by reading the bunch of line provided by:

cat /proc/drbd

DRBD monitoring is a must doing so you can ensure you’re data is consistent and uptodate. I worked with DRBD 8.4 and tried to figure out something that check the whole DRBD status.

I mean the folloing by conserving the order:

  1. Split brain
  2. Connection status
  3. Ressource Role
  4. Disk status
  5. I/O status
  6. Performance

I’m no DRBD expert, I just tried think something logical to my monitoring would be efficient and include no redundant alarm. And yes, we, mathematicians, adore optimization 🙂 :).

The flow of checks goes like in the following chart:

monitor drdbd logic