Implementing ls command in C

We know that ls command in unix displays the list of files in a directory. It has various options to display the data in various styles and formats.

By default its implementation is part of Coreutils package, which is default package in all Linux flavours.

I want to see how we can implement its basic functionality with a simple C program.

Decoding hardware information of a PC

Checking out machine hardware information is no more a geeky thing of olden days where you need to go through the hardware specification documents or to open up a physical machine to find out the hardware details if the specification documents are missing. Now we have some really cool handy software tools to help us out.

I thought to make a page with some important tools which help us in decoding the hardware related information on a Linux Box. Feel free to drop a comment with the tool names I missed out in this post.

NOTE: All of them are ordered in alphabetical order and I am trying these tools on a Ubuntu machine running in virtual box. So some of the tool outputs might be displaying names likes VirtualBox and Oracle Corporation.

Uli Drepper - Basics you should know about buffer overflow

Part 1: Buffer Overflows

Connecting via SSH without password prompt

SSH is the common way to connect remote machines these days. Almost all kinds of applications use SSH in background for communicating with end machines.

But sometimes typing password repeatedly for establishing a SSH connection with the trusted end machine is quite daunting task.

Let us see how to automate the things in 3 simple steps where we can ssh to "user@host" without asking a password every time. Replace user with the username and host with the remote machine ip-address or hostname. For E.g. john@192.168.245.129

NOTE: Only do this with trusted machines.

Daemon-izing a Process in Linux

A Linux process works either in foreground or background.

A process running in foreground can interact with the user in front of the terminal. To run a.out in foreground we execute as shown below.

./a.out

When a process runs as background process then it runs by itself without any user interaction. The user can check its status but he doesn't (need to) know what it is doing. To run a.out in background we execute as shown below.

$ ./a.out &
[1] 3665

As shown above when we run a process with & at the end then the process runs in background and returns the process id (3665 in above example).

what is a DAEMON Process?
A 'daemon' process is a process that runs in background, begins execution at startup
(not neccessarily), runs forever, usually do not die or get restarted, waits for requests to arrive and respond to them and frequently spawn other processes to handle these requests.

So running a process in BACKGROUND with a while loop logic in code to loop forever makes a Daemon ? Yes and also No. But there are certain things to be considered when we create a daemon process. We follow a step-by-step procedure as shown below to create a daemon process.

1. Create a separate child process - fork() it.
Using fork() system call create a copy of our process(child), then let the parent process exit. Once the parent process exits the Orphaned child process will become the child of init process (this is the initial system process, in other words the parent of all processes). As a result our process will be completely detached from its parent and start operating in background.

pid=fork();

if (pid<0) exit(1); /* fork error */

if (pid>0) exit(0); /* parent exits */

/* child (daemon) continues */

Notification Chains in Linux Kernel - Part 03

Continuation after PART-2.


Notifying Events on a Chain 
Notifications are generated with notifier_call_chain. This function simply invokes, in order of priority, all the callback routines registered against the chain. Note that callback routines are executed in the context of the process that calls notifier_call_chain. A callback routine could, however, be implemented so that it queues the notification somewhere and wakes up a process that will look at it.

NOTE: Similar to register and unregister functions we don't directly call notifier_call_chain function as we have wrapper functions for respective chains.

 <kernel/notifier.c>

 58 static int __kprobes notifier_call_chain(struct notifier_block **nl,
 59                     unsigned long val, void *v,
 60                     int nr_to_call, int *nr_calls)
 61 {
 62     int ret = NOTIFY_DONE;
 63     struct notifier_block *nb, *next_nb;
 64 
 65     nb = rcu_dereference(*nl);
 66 
 67     while (nb && nr_to_call) {
 68         next_nb = rcu_dereference(nb->next);
 69         ret = nb->notifier_call(nb, val, v);
 .
 76         nb = next_nb;
 77         nr_to_call--;
 78     }
 79     return ret;
 80 }

• nl
  Notification chain. 


• val
  Event type. The chain itself identifies a class of events; val unequivocally identifies an event type (i.e., NETDEV_REGISTER). 


• v
  Input parameter that can be used by the handlers registered by the various clients. This can be used in different ways under different circumstances. For instance, when a new network device is registered with the kernel, the associated notification uses v to identify the net_device data structure.


• nr_to_call
  Number of notifier functions to be called. Don't care value of this parameter is -1. 


• nr_calls
  Records the number of notifications sent. Don't care value of this field is NULL.

Notification Chains in Linux Kernel - Part 02

Continuation after PART-1.

Check the PART-3. 

Blocking Notifier chains
A blocking notifier chain runs in the process context. The calls in the notification list could be blocked as it runs in the process context. Notifications that are not highly time critical could use blocking notifier chains.

Linux modules use blocking notifier chains to inform the modules on a change in QOS value or the addition of a new device.

<kernel/notifier.c>

186 int blocking_notifier_chain_register(struct blocking_notifier_head *nh,
187         struct notifier_block *n)
188 {
.
199     down_write(&nh->rwsem);
200     ret = notifier_chain_register(&nh->head, n);
201     up_write(&nh->rwsem);
202     return ret;
203 }
204 EXPORT_SYMBOL_GPL(blocking_notifier_chain_register)
.
216 int blocking_notifier_chain_unregister(struct blocking_notifier_head *nh,
217         struct notifier_block *n)
218 {
.
229     down_write(&nh->rwsem);
230     ret = notifier_chain_unregister(&nh->head, n);
231     up_write(&nh->rwsem);
232     return ret;
233 }
234 EXPORT_SYMBOL_GPL(blocking_notifier_chain_unregister);

Notification Chains in Linux Kernel - Part 01

Linux is a monolithic kernel. Its subsystems or modules help to keep the kernel light by being flexible enough to load and unload at runtime. In most cases, the kernel modules are interconnected to one another. An event captured by a certain module might be of interest to another module.

Typically, communication systems implement request-reply messaging, or polling. In such models, a program that receives a request will have to send the data available since the last transaction. Such methods sometimes require high bandwidth or they waste polling cycles.

To fulfill the need for interaction, Linux uses so called notification chains. These notifier chains work in a Publish-Subscribe model. This model is more effective when compared to polling or the request-reply model.

For each notification chain there is a passive side (the notified) and an active side (the notifier), as in the so-called publish-and-subscribe model:

• The notified are the subsystems that ask to be notified about the event and that provide a callback function to invoke.

• The notifier is the subsystem that experiences an event and calls the callback function.

NOTE: All the code samples are taken from Linux 2.6.24 kernel.

struct notifier_block
The elements of the notification chain's list are of type notifier_block:

<include/linux/notifier.h>

 50 struct notifier_block {
 51     int (*notifier_call)
(struct notifier_block *, unsigned long, void *);
 52     struct notifier_block *next;
 53     int priority;
 54 };

• notifier_call - function to execute. 

• next - used to link together the elements of the list.

• priority - the priority of the function. Functions with higher priority are executed first. But in practice, almost all registrations leave the priority out of the notifier_block definition, which means it gets the default value of 0 and execution order ends up depending only on the registration order (i.e., it is a semirandom order).

The notifier_block data structure is a simple linked list of function pointers. The function pointers are registered with ‘functions’ that are to be called when an event occurs. Each module needs to maintain a notifier list. The functions are registered to this notification list. The notification module (publisher) maintains a list head that is used to manage and traverse the notifier block list. The function that subscribes to a module is added to the head of the module’s list by using the register_xxxxxx_notifier API and deletion from the list is done using unregister_xxxxxx_notifier.

strace - diagnostic, debugging and reverse engineering tool

Many times we come across hopeless situations where a program when compiled and installed in GNU/Linux just fails to run. Then we have to trace the output of the misbehaving program. But tracing the output of a program throws up a lot of data and it is a daunting task to go through volumes of data. Still there are cases where we are not fruitful in pin pointing the cause of error.

In this situation strace also known as system-call tracer comes for rescue. It is a debugging tool that monitors the system calls used by a program and all the signals it receives.

A system call is the most common way programs communicate with the kernel. System calls include reading and writing data, opening and closing files and all kinds of network communication. Under Linux, a system call is done by calling a special interrupt with the number of the system call and its parameters stored in the CPU's registers.

Using strace is quite simple. There are two ways to let strace monitor a program.

Method 1:

To start strace along with a program, just run the executable with strace as shown below.

strace program-name

For example let us trace ls command.

$ strace ls
execve("/bin/ls", ["ls"], [/* 39 vars */]) = 0
brk(0)                                  = 0x82d4000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7787000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=76503, ...}) = 0
mmap2(NULL, 76503, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7774000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libselinux.so.1", O_RDONLY)  = 3
read(3, "177ELF111���������3�3�1���@G��004���"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=104148, ...}) = 0
mmap2(NULL, 109432, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x41d000
mmap2(0x436000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18) = 0x436000
close(3)                                = 0
.
.
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7613000
write(1, "01.c  a.outn", 1201.c  a.out
)           = 12
close(1)                                = 0
munmap(0xb7613000, 4096)                = 0
close(2)                                = 0
exit_group(0)                           = ?

In the above example we are not displaying the complete output of strace command. Even though output from strace looks very complicated, this is only due to many system calls made when loading shared libraries. However, once we have found which system calls are the important ones (mainly open, read, write and the like), the results will look fairly intuitive to us.

Method 2:

If we want to monitor a process which is currently running we can attach to the process using –p option. Thus we can even debug a daemon process.

strace –p <pid-of-the-application>

For e.g

#include <stdio.h>
#include <unistd.h>

int main()
{
   sleep(20);
   return 0;
}

We will compile the above code and run it as a background process. Then we try to monitor the program using its process id as shown below.

$ gcc main.c

$ ./a.out &
[1] 1885

$ strace -p 1885
Process 1885 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
exit_group(0)                           = ?
Process 1885 detached
[1]+  Done                    ./a.out

In contrast to a debugger, strace does not need a program's source code to produce human-readable output.

Some handy options

Below example is used in the discussion of other important options supported by strace.

#include <stdio.h>

int main(void)
{
  FILE *fd = NULL;

  if(fd = fopen("test","rw"))
    {   
      printf("TEST file openedn");
      fclose(fd);
    }   
  else
    {   
      printf("Failed to open the filen");
    }   

  return 0;
}

Providing the time taken by multiple system calls in a program

Using –c option strace provides summary information on executing a program.

It provides information like number of times a system call is used, time spent executing various system calls, number of times errors returned as shown below.

$ strace -c ./a.out 
Failed to open the file
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 91.47    0.004000        4000         1           execve
  8.53    0.000373         124         3         3 access
  0.00    0.000000           0         1           read
  0.00    0.000000           0         1           write
  0.00    0.000000           0         3         1 open
  0.00    0.000000           0         2           close
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         3           mprotect
  0.00    0.000000           0         7           mmap2
  0.00    0.000000           0         3           fstat64
  0.00    0.000000           0         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.004373                    29         4 total

Redirecting the output to a file 

Using -o option we can redirect the complex output of strace into a file.

$ strace -o <output-file-name> <program-name>

Time spent per system call 

Using –T option we can get time spent per system call. In the below example we can see time spent per system call is printed at the end of the line.

$ strace -T ./a.out 
execve("./a.out", ["./a.out"], [/* 39 vars */]) = 0 <0.003256>
.
brk(0x9db0000)                          = 0x9db0000 <0.000123>
open("test", O_RDONLY)                  = -1 ENOENT (No such file or directory) <0.000154>
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 <0.000125>
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77d4000 <0.000121>
write(1, "Failed to open the filen", 24Failed to open the file
) = 24 <0.000258>
exit_group(0)                           = ?

Prefixing time of the day for every line in trace 

It is useful sometimes to track at what time a particular is triggered. By using -t option strace will prefix each line of the trace with the time of day, which will be really helpful to find out at particular time at which call is the process blocked.

$ strace -t ./a.out 
execve("./a.out", ["./a.out"], [/* 39 vars */]) = 0 <0.003256>
.
brk(0x9db0000)                          = 0x9db0000 <0.000123>
open("test", O_RDONLY)                  = -1 ENOENT (No such file or directory) <0.000154>
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 <0.000125>
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77d4000 <0.000121>
write(1, "Failed to open the filen", 24Failed to open the file
) = 24 <0.000258>
exit_group(0)                           = ?

Tracing only specific system calls 

Using –e option we can also specify which system calls to be traced. To trace only open() and close() system calls use the following command:

$ strace –e trace=’open,close’ <program-name>

Similarly we can also use negation option to not trace specific system calls. If we don’t want to trace open() system call in previous example we can give the below command

$ strace -e trace='!open,close' ./a.out

Check the man page of strace for other options.

Fedora 12 on HP Pavilion dv6340eu

Atlast the day has come, LINUX running on my laptop(pre-loaded with M$ VISTA). Time for celebrations.

So I decided to drop a post on how the journey took place towards installing and using LINUX in my laptop. Initially I faced a serious problem, which LINUX flavour to use ... UBUNTU or FEDORA (sorry these are only my preferred flavours for long time). I got UBUNTU successfully running on my old desktop, on my sister's latest desktop and some more desktops of friends. I got FEDORA running successfully on my SERVER and other desktops. I never tried or used LINUX flavours on a LAPTOP (as I see communities filled with posts saying problems with LINUX on their laptops) so I have to choose one ...

Atlast I decided to go with FEDORA 12 .. not sure why I dumped UBUNTU over FEDORA ... but since I am comfortable with FEDORA as I work more with FEDORA .. and FEDORA 12 got many new open-source technologies and pre-packed with latest kernel in the market (Hmm I agree that a UBUNTU release is 6 months older to a FEDORA release). But still I love UBUNTU :)

Laptop Specs
HP Pavilion dv6340eu (2007 European model)
AMD Turion 64 X2 processor,
1GB RAM (really insane),
NVIDIA GEOFORCE 7200M graphics card,
BROADCOM wireless chipset and other HP stuff.

Off the Road ... M$ VISTA
Boot up and complete M$ VISTA loading takes around 10 to 15 minutes (low RAM might be the cause) until then don't even think of opening any other application it will mess up. Sometimes I use a NOKIA mobile to connect to WIRELESS GPRS INTERNET of AIRTEL and it requires PC-SUITE software from NOKIA. Apart from this inorder to protect myself from viruses and online threats I need a third party security application(s) like ANTIVIRUS, FIREWALL (resource hogs on 1GB RAM machine running M$ VISTA). Even in idle state more than 60% of the RAM is always busy and processor(s) utilization is always more than 50%. I faced strange problems on M$ VISTA with External harddisks .. when I use "safely remove hardware" option and then disconnect the hardware it always pop up message saying some information might be lost ... eventhough I wait for sometime to disconnect them?? and it really messed up my external storage devices many times.

Back to the Road ...
I decided to go with a 32 bit FEDORA OS installation ... after seeing many problems reported with 64 bit OS when compared to 32 bit OS. Downloaded a LIVECD and installed FEDORA 12 in dual boot without any hiccups ... that's a plain install no magic spells or hacking cheats used :). Voilla great its really simple on my laptop.

Automatic reboot of laptop once the installation is complete and booted in FEDORA12 ... Wow it just took less than 2 minutes to boot up completely ... RAM and CPU usage has been drastically reduced.

I connected my NOKIA mobile to laptop and it is detected automatically. I created a new wireless broadband connection under my subscriber AIRTEL and it simply connected like a charm. Damn who needs NOKIA PC-SUITE crap for connecting to internet. I am not using any of the fancy COMPIZ stuff and simply using the NOUVEAU drivers for my graphic cards. My wirelan connection is also working like a charm no drivers from HP or any third party stuff.

Problems in Journey ..
Small usage problems with openoffice ... as my brain is corrupted with M$ OFFICE. Eventhough LIVE CD is downloaded recently it has got many security updates and patches (around 412 I guess .. of 300MB) to be downloaded from internet ... I wonder why don't they update the FEDORA build atleast once in a month with latest patches :(. Livecd doesn't have any compilers for C, C++, Java and so on and they have to be updated from internet :(. Fedora by default has music/movie players but none of them have any codecs to play a normal MP3 file (hmm licensing problems)...

Seems there are some problems with power management .. as my laptop makes me feel as if it is burning ... My harddisk temperature is at 60'c and processor cores are above 65'c eventhough internal fans are running at full speed.

After all the problems and happiness I decided to drop this post in BLOGGER from a LINUX machine ... my FIRST one. Now my laptop proudly carry the logo of FEDORA on its chest.


Update 1: Do check Fedora 12 common bugs before you install Fedora 12 on your machine - https://fedoraproject.org/wiki/Common_F12_bugs

Update 2: Integrated Webcam is still not working .. looking for workarounds.

Update 3: Do not use Nvidia Proprietary drivers replacing NOUVEAU drivers. Atleast I am seeing a system crash after update. Raised a bug at https://bugzilla.redhat.com/show_bug.cgi?id=594439 and waiting for the real reason behind the crash and any workarounds.

Hmm ... haven't found any solution until now ... Only solution available is REINSTALL OS again ... stupid workaround. Why should I stuck with FEDORA 12 which seems to be crappy .. UBUNTU 10.04 with 5 years of support here I come.

Update 4:BYE BYE FEDORA 12 .... I am really disappointed by the support and help provided by Fedora Community.

iptables - Rate-limit incoming connections

Using "recent" match option in iptables we can match recent connections, and perform simple throttling operation on incoming connections i.e. we can create simple firewall rules which will deny access from remote clients who attempt to connect "too many" times.

"recent" dynamically creates a list of IP addresses and then match against that list. The functionality of "recent" match option is simple, a list of IP addresses are created dynamically, which can be used in the future to test connection attempts against.

rate limiting is conceptually different from bandwidth throttling/limiting; a bandwidth-throttled connection will queue packets and limit the rate at which they are transmitted/received. Rate limiting will not do this; when you use rate limiting on, for example, incoming TCP connection attempts to your identd, and connections exceeding the specified limit will be denied; there is no queueing of packets.

NOTE: CONFIG_IP_NF_MATCH_RECENT flag should be enabled in Linux Kernel to use "recent" option.

Following 2 rules will limit incoming connections to destination port 22 not more than 3 in a minute and more than that will be dropped:

iptables -I INPUT -p tcp --dport 22 -m state --state NEW \
-m recent --set

iptables -I INPUT -p tcp --dport 22 -m state --state NEW \
-m recent --update --seconds 60 --hitcount 3 -j DROP

"--state NEW" – To make sure that only new connections are managed.

"--set" flag will make sure that the IP address of the host which initiated the connection will be added to the "recent list", where it can be tested and used again in the future i.e. in our second rule.

The second rule is where the actual magic happens.

"—update" flag tests whether the source IP address is in the list of recent connections, in our case each new tcp connection to destination port 22 will be in the list because we used the "--set" flag to add it in the preceeding rule.

"--seconds" flag is used to make sure that the source IP address is only going to match if the last connection was within the timeframe given.

"--hitcount" flag matches only if the given count of connection attempts is greater than or equal to the number given.

The second rule will DROP an incoming connection if:

• The IP address which initiated the connection has previously been added to the list and
• The IP address has sent a packet in the past 60 seconds and
• The IP address has sent more than 3 packets in total.

Let use see another example where we try to limit ICMP echo requests to not more than 30 in a minute:

iptables -I INPUT -p icmp --icmp-type echo-request \
-m recent --set

iptables -I INPUT -p icmp --icmp-type echo-request \
-m recent --update --seconds 60 --hitcount 30 -j DROP

In above rules we dont use "--state NEW" since it is not needed in rate limiting number of ICMP echo requests.

NOTE: In the above rules we are trying to automatically limit number of connections from each user. So it is something like 3 attempts from each user in 1 minute.

What if we want to limit number of certain packets from all users ? Then "limit" match option comes to rescue.

"limit" option specifies the maximum average number of matches to allow per second. We can specify time intervals in the format /second, /minute, /hour, or /day, or you can use abbreviations so that 3/second is the same as 3/s.

NOTE: CONFIG_IP_NF_MATCH_LIMIT flag should be enabled in Linux Kernel to use "limit" option.

Let us see how to limit number of ICMP echo requests not more than 3 per second and drop rest of them.

iptables -I INPUT -p icmp --icmp-type echo-request \
-m limit !--limit 3/s -j ACCEPT

When tuned correctly, this feature allows us to filter unusually high volumes of traffic that characterize denial of service (DOS) attacks and Internet worms.

But take care: When tuned incorrectly, this feature does the opposite: Helping any attacker in denial of service attacks. Instead of having to initiate enough connections to bring the whole server down, it then may be sufficient to just start enough connections to activate the firewall rules.

References:
1. Debian Administration
2. man iptables

Creating and using static libraries in Linux

Static libraries are simply a collection of ordinary object files.

For more information on shared libraries checkout - Creating and using shared libraries in Linux

Static libraries conventionally end with the ".a" suffix. They are useful for developers to link to their library, but don't want to give the library source code. Theoretically code in static ELF libraries that is linked into an executable should run slightly faster (by 1-5%) than a shared library or a dynamically loaded library, but in practice this rarely seems to be the case due to other confounding factors.

We use following source code files for this post.

calc_mean.c

double mean(double a, double b)
{
return (a+b) / 2;
}

calc_mean.h

double mean(double, double);

main.c - We are including our static library in this application.

#include <stdio.h>
#include "calc_mean.h"

int main(int argc, char* argv[]) {

double v1, v2, m;
v1 = 5.2;
v2 = 7.9;

m  = mean(v1, v2);

printf("The mean of %3.2f and %3.2f is %3.2f\n", v1, v2, m);

return 0;
}

Creating the static library

First we have to create object file for calc_mean.c

gcc -c calc_mean.c -o calc_mean.o

Then, using archiver (ar) we produce a static library (named libmean.a) out of the object file calc_mean.o.

ar rcs libmean.a calc_mean.o

NOTE: A static library must start with the three letters 'lib' and have the suffix '.a'.

Compiling main program and linking with static library

We have already created a static library libmean.a and now let us use the static library by invoking it as part of the compilation and linking process when creating a program executable. Incase of gcc we use following flags to create static library

• -llibrary
searches for the library named library at the time of linking. Linker searches and processes libraries and object files in the order they are specified. Thus, ‘foo.o -lz bar.o’ searches library ‘z’ after file foo.o but before bar.o. If bar.o refers to functions in ‘z’, those functions may not be loaded. The linker searches a standard list of directories for the library, which is actually a file named liblibrary.a. The linker then uses this file as if it had been specified precisely by name.

• -L(path-to-the-library)
Specifies the path in which the library file can be found. We can use -L. inorder to point to the current directory and -L/home/tmp to point to /home/tmp directory.

• -static
On systems that support dynamic linking, this prevents linking with the shared libraries.

gcc -static main.c -L. -lmean -o main

Now run the executable program 'main'

$./main

Creating Shared Libraries in Linux - Part 2

Check the PART1 of this article.

4. Making the library available at run-time

Using LD_LIBRARY_PATH
We have to create or set the environment variable "LD_LIBRARY_PATH" to the directory containing the shared libraries.

export LD_LIBRARY_PATH=/home/cf/lib

If in current directory you can give the following command

export LD_LIBRARY_PATH=.

If we have to append a new directory to the existing paths then add the directories separated by colons to environment variable "LD_LIBRARY_PATH".

export LD_LIBRARY_PATH=/opt/lib:$LD_LIBRARY_PATH

Now recompile the main program

gcc -o test main.c -lcalc_mean -L/home/cf/slib

Now check the ldd command output

$ ldd test
 linux-gate.so.1 =>  (0x007ad000)
 libcalc_mean.so => ./libcalc_mean.so (0x0081e000)
 libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x005ff000)
 /lib/ld-linux.so.2 (0x00e19000)

It seems now linker is able to locate our shared library as we can see in above output. Now run the program

$./test

LD_LIBRARY_PATH is good for quick tests and for systems on which you don’t have admin privileges. As a downside, however, exporting the LD_LIBRARY_PATH environment variable  might screw up with other programs you run that also rely on LD_LIBRARY_PATH if you don’t reset it to its previous state when you’re done.

Creating and using shared libraries in Linux

What is a Shared Library ??

A library that is loaded into physical memory only once and reused by multiple processes via virtual memory. 

Generally Shared libraries are .so (or in windows .dll) files.

Why shared libraries ??

• They reduce memory consumption if used by more than one process, and they reduce the size of the executable.
• They make developing applications easier: a small change in the implementation of a function in the library don't need the user to recompile and relink his application code every time. You need to only relink if you make incompatible changes, such as adding arguments to a call or changing the size of a struct.

NOTE: Debugging using a shared library is slightly more difficult when compared with static libraries, because the debugger usually used on Linux, gdb, has some problems with shared libraries.

Let us see how to create a shared library on Linux. We use following source code files for this post.
calc_mean.h

#ifndef calc_mean_h__
#define calc_mean_h__
double mean(double, double);
#endif  // calc_mean_h__

calc_mean.c

double mean(double a, double b)
{
return (a+b) / 2;
}

main.c - We are including our shared library in this application.

#include <stdio.h>
#include "calc_mean.h"

int main(int argc, char* argv[]) {

double v1, v2, m;
v1 = 5.2;
v2 = 7.9;

m  = mean(v1, v2);

printf("The mean of %3.2f and %3.2f is %3.2f\n", v1, v2, m);

return 0;
}

1. Creating Object File with Position Independent Code
All the code that goes into a shared library needs to be position independent. We can make gcc emit position-independent code by passing it one of the command-line switches -fpic or -fPIC (the former is preferred, unless the modules have grown so large that the relocatable code table is simply too small in which case the compiler will emit an error message, and you have to use -fPIC).

First we will create object files for all .c files that goes into a shared library.

gcc -c -fPIC calc_mean.c -o calc_mean.o

Above we are compiling calc_mean.c with -fPIC option and generating calc_mean.o object file.

2. Creating Shared library with the Object File
Every shared library has a prefix "lib", the name of the library, the phrase ".so", followed by a period and a version number that is incremented whenever the interface changes (as a special exception, the lowest-level C libraries don't start with "lib").

gcc -shared -o libcalc_mean.so calc_mean.o

Above command on successful produces a shared library named "libcalc_mean.so".

• -shared: Produces a shared object which can then be linked with other objects to form an executable.

3. Using the Shared Library
Now let us link the created shared library with our application. Compile main.c as shown below

$ gcc -o test main.c -lcalc_mean
/usr/bin/ld: cannot find -lcalc_mean
collect2: ld returned 1 exit status

The linker doesn’t know where to find libcalc_mean. But why ?
GCC has a list of places to look by default for shared libraries, but our directory is not in that list. Bingo that's the reason compilation failed at linking level.

Now we need to tell GCC where to find libcalc_mean.so. We will do that with the -L option.

gcc -o test main.c -lcalc_mean -L/home/cf/slib

•  -l option tells the compiler to look for a file named libsomething.so The something is specified by the argument immediately following the “-l”. i.e. -lmean

• -L option tells the compiler where to find the library. The path to the directory containing the shared libraries is followed by "-L". If no “-L” is specified, the compiler will search the usual locations. "-L." means looking for the shared libraries in the current directory and "-L/home/cf/lib" means looking for the shared libraries at "/opt/lib" path. You can specify as many “-l” and “-L” options as you like.

NOTE: It would be a better idea to move all personal shared libraries in one directory rather in the current directory. For easy understanding I am moving "libcalc_mean.so" to "/home/cf/slib".

mv libcalc_mean.so /home/cf/slib

Now compile main.c. It would be successful and creates an executable named "test".
Let us check if the path to our shared library is included successfully into the executable by linker as shown below.

ldd executablename

$ ldd test 
 linux-gate.so.1 =>  (0x00332000)
 libcalc_mean.so => not found
 libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x006aa000)
 /lib/ld-linux.so.2 (0x00db9000)

You can see that linker cannot find our shared library libcalc_mean.so.

Basically libraries are present in /usr/lib amongst other places. Static libraries (.a suffix) are incorporated into the binary at link time, whereas dynamic ones (.so suffix) are referenced by location.

Check the PART2 of this article to understand further

Using 'find' command

'find' is one of the useful commands available in Unix/Linux systems. In this article I am trying to use 'find' in a effective way.

Basic syntax of find command as per man page is - find [path...] [expression]

NOTE: There are many options available for find command which are available in man page for find. Only some of those options are shown in this tutorial.

Finding a particular file in your system

$ find / -name 'filename' 2>/dev/null
$ find / -name 'filename' 2>errors.txt

/ - Start searching from the root directory (i.e / directory)

-name - Given search text is the filename rather than any other attribute of a file 'filename'. Always enclose the filename in single quotes...

NOTE: 2>/dev/null is not related to find tool as such. 2 indicates the error stream in Linux, and /dev/null is the device where anything you send simply disappears. So 2>/dev/null in this case means that while finding for the files, in case any error messages pop up simply send them to /dev/null i.e. simply discard all error messages.

Alternatively you could use 2>error.txt where after the search is completed you would have a file named error.txt in the current directory with all the error messages in it.

$ find -name 'met*'

The above command would start searching for the files that begin with the letters 'met' within the current directory and the directories that are present within the current directory.

If no paths are given, the current directory is used. If no expression is given, the expression ‘-print’ is used.

Searching with respect to type of the file (-type)

$find . -name 'temp'
./keepout/temp
./temp

$find . -name 'temp' -type d
./temp

Where d - directory, p - named pipe (FIFO), f - regular file and so on

Ignoring case-sensitivity (-iname)

$ find /home/temp -iname 'index*'

This command searchs for a file starting with string 'index' without considering the case of the filename. So all files starting with any combination of letters in upper and lower case such as INDEX or indEX or index would be returned.

Searching for a file based on size and time

$ find /home/songs -name '*.mp3' -size -5000k

$ find / -size +10000k

First command finds within a directory called /home/songs, only those mp3 files that have a size less than 5000 Kilobytes.

$ find /home/temp -amin -10 -name '*.c'
$ find /home/temp -atime -2 -name '*.c'
$ find /home/temp -mmin -10 -name '*.c'
$ find /home/temp -mtime -2 -name '*.c'

The 1st command searches for those files that are present in the directory /home/temp and its subdirectories which end in .c and which have been accessed in the last 10 minutes.

The 2nd command does the same but searches for those files that have been accessed in the last 10 hours.

The 3rd and the 4th commands do the same as the 1st and 2nd commands but they search for modified files rather than accessed files. Only if the contents of the files have been modified, would their names be returned in the search results.

$ find / -mount -name 'win*'

This command searches for files starting with the letters 'win' in their filenames. The only difference is that the mounted filesystems would not be searched for this time. This is useful when you have your Windows partitions mounted by default. And a search for 'win' might return many files on those partitions, which you may not be really interested in. This is only one use of -mount parameter.

$ find /home/songs -name 'Metallica*' -and -size +10000k
$ find /home/songs -size +10000k ! -name "Metallica*"
$ find /home/songs -name 'Metallica*' -or -size +10000k

Boolean operators such as AND, OR and NOT make find an extremely useful tool.

The 1st command searches within the directory /songs for files that have their names beginning with 'Metallica' and whose size is greater than 10000 kilobytes (> 10 MB).

The 2nd command searches in the same directory as above case but only for files that are greater than 10MB, but they should not have 'Metallica' as the starting of their filenames.

The 3rd command searches in the same directory for files that begin with 'Metallica' in their names or all the files that are greater than 10 MB in size.

How to apply a unix command to a set of files (-exec) ?

$ find . -name '*.sh' -exec chmod o+r '{}' \; -print

This command will search in the current directory and all sub directories. All files ending with .sh extension will be processed by the chmod -o+r command. The argument '{}' inserts each found file into the chmod command line. The \; argument indicates the exec command line has ended.

The end results of this command is all .sh files have the other permissions set to read access (if the operator is the owner of the file).

Searching for a string in a selection of files (-exec grep ...).

$ find . -exec grep "hello" '{}' \; -print

Prints all files that contain the string 'hello' will have their path printed to standard output.

If you want to just find each file then pass it on for processing use the -q grep option. This finds the first occurrance of the search string. It then signals success to find and find continues searching for more files.

find . -exec grep -q "hello" '{}' \; -print

Finding Empty files (-empty)

$find . -empty

To delete empty files in the current directory:

$ find . -empty -maxdepth 1 -exec rm '{}' \;

For more examples try out

1. linux.ie
2. Devdaily
3. hccfl.edu

Signals in Linux - Blocking Signals

Blocking a signal means telling the operating system to hold it and deliver it later when it is unblocked. Between the time when it is generated and when it is delivered a signal is said to be pending.

Generally, a program does not block signals indefinitely - it might as well ignore them by setting their actions to SIG_IGN. But it is useful to block signals briefly, to prevent them from interrupting sensitive operations.

Is Blocking a signal similar to Ignoring a signal ?

No, blocking a signal is different from ignoring a signal. When a process blocks a signal, the operating system does not deliver the signal until the process unblocks the signal. A process blocks a signal by modifying its signal mask with sigprocmask. But when a process ignores a signal, the signal is delivered and the process handles it by throwing it away.

How Blocking Signals is Useful ?

Temporary blocking of signals with sigprocmask gives you a way to prevent interrupts during critical parts of your code. If signals arrive in that part of the program, they are delivered later, after you unblock them.

One example where this is useful is for sharing data between a signal handler and the rest of the program. If the type of the data is not sig_atomic_t, then the signal handler could run when the rest of the program has only half finished reading or writing the data. This would lead to confusing consequences.

To make the program reliable, you can prevent the signal handler from running while the rest of the program is examining or modifying that data - by blocking the appropriate signal around the parts of the program that touch the data. Blocking signals is also necessary when you want to perform a certain action only if a signal has not arrived.

All signal blocking functions use a data structure called a signal set to specify what signals are affected. Thus, every activity involves two stages: creating the signal set, and then passing it as an argument to a library function. These facilities are declared in the header file signal.h.

The sigset_t data type is used to represent a signal set. Internally, it may be implemented as either an integer or structure type. For portability, use only the functions described below to initialize, change, and retrieve information from sigset_t objects - don't try to manipulate them directly.

#include <signal.h>

int sigemptyset(sigset_t *set);

int sigfillset(sigset_t *set);

int sigaddset(sigset_t *set, int signum);

int sigdelset(sigset_t *set, int signum);

int sigismember(const sigset_t *set, int signum);

sigemptyset function initializes the signal set given by set to empty, with all signals excluded from the set.

sigfillset function initializes set to full, including all signals.

sigaddset and sigdelset functions add and delete respectively signal signum from set.

sigismember function tests whether signum is a member of set.

Objects of type sigset_t must be initialized by a call to either sigemptyset or sigfillset before being passed to the functions sigaddset, sigdelset and sigismember.

For more information checkout: man 3 sigsetops

The collection of signals that are currently blocked is called the signal mask. Each process has its own signal mask. When you create a new process, it inherits its parent's mask. You can block or unblock signals with total flexibility by modifying the signal mask.

#include <signal.h>

int sigprocmask(int how, const sigset_t *set, sigset_t *oldset);

In a traditional single-threaded application, sigprocmask system call can be used to fetch and/or manipulate the signal mask of the calling thread.

how determines what operation to be performed on the signal mask.
If oldset is non-null, the previous value of the signal mask is stored in oldset.
set determines list of signals to be set in blocking state.

Signals, such as SIGSTOP and SIGKILL, cannot be blocked. If an attempt is made to block these signals, the system ignores the request without reporting an error.

NOTE: Do not use sigprocmask in multi-threaded processes, because each thread has its own signal mask and there is no single process signal mask. According to POSIX, the behavior of sigprocmask in a multi-threaded process is "unspecified". Instead, use pthread_sigmask.

For more information checkout: man 2 sigprocmask

In the below example we try to block and unblock the SIGINT signal continually in a loop. If a user enters Ctrl-C while SIGINT is blocked, then the program terminates only after it is unblocked. If a user types Ctrl-C while SIGINT is unblocked, the program terminates immediately.

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc,  char *argv[]) {
int i;
sigset_t intmask;
int repeatfactor;
double y = 0.0;

if ((sigemptyset(&intmask) == -1) || (sigaddset(&intmask, SIGINT) == -1)){
   perror("Failed to initialize the signal mask");
   return 1;
}

for ( ; ; ) {
   printf("Entering BLOCK state\n");
   if (sigprocmask(SIG_BLOCK, &intmask, NULL) == -1)
       break;
   fprintf(stderr, "SIGINT signal blocked\n");
   sleep(2);

   printf("Leaving Blocking State & Entering UNBLOCK state\n");
   if (sigprocmask(SIG_UNBLOCK, &intmask, NULL) == -1)
       break;
   fprintf(stderr, "SIGINT signal unblocked\n");
   sleep(2);
}
perror("Failed to change signal mask");
return 1;
}


Output:

$ ./a.out
Entering BLOCK state
SIGINT signal blocked
Leaving Blocking State & Entering UNBLOCK state
SIGINT signal unblocked
^C

$ ./a.out
Entering BLOCK state
SIGINT signal blocked
^CLeaving Blocking State & Entering UNBLOCK state

$

Signals in Linux - Catching and Ignoring Signals - sigaction

The sigaction system call has the same basic effect as signal system call: to specify how a signal should be handled by the process.

But sigaction offers more control over the signal mechanism, at the expense of more complexity. In particular, sigaction allows you to specify additional flags to control when the signal is generated and how the handler is invoked.

Hence for reference we can say that sigaction is more like opensource Linux OS flavours more options, more control and complex for normal users than the proprietary stuff.

#include <signal.h>

int sigaction(int signum, const struct sigaction *action,struct sigaction *oldaction);

signum specifies the signal and can be any valid signal except SIGKILL and SIGSTOP.

The action argument is used to set up a new action for the signal signum, while the oldaction argument is used to return information about the action previously associated with this symbol. If action is non-null, the new action for signal signum is installed from action. If oldaction is non-null, the previous action is saved in oldaction.

For more information checkout: man 2 sigaction

struct sigaction {
 void     (*sa_handler)(int);   /*SIG_DFL, SIG_IGN, or
                                a function pointer*/
 void     (*sa_sigaction)(int, siginfo_t *, void *);
 sigset_t   sa_mask;
 int        sa_flags;
 void     (*sa_restorer)(void);
};

Portability Note: The basic signal function is a feature of ISO C, while sigaction is part of the POSIX.1 standard. If you are concerned about portability to non-POSIX systems, then you should use the signal function instead.

static volatile sig_atomic_t doneflag = 0;

int main (void) {
 struct sigaction act;

 act.sa_handler = setdoneflag;/* set up signal handler */
 act.sa_flags = 0;
 if ((sigemptyset(&act.sa_mask) == -1) ||
       (sigaction(SIGINT, &act, NULL) == -1)) {
     perror("Failed to set SIGINT handler");
     return 1;
 }

 while (!doneflag) {
     printf("press CTRL+C to kill the loop\n");
     sleep(1);
 }

 printf("Program terminating ...\n");
 return 0;
}

Output:
$ ./a.out
press CTRL+C to kill the Loop
press CTRL+C to kill the Loop
^C

In SignalHandler - setdoneflag
Program terminating ...

sig_atomic_t is an integer type that is guaranteed by the standard to not be partially written or partially read in the presence of asynchronous interrupts.

If we set act.sa_handler as SIG_DFL then the program simply terminates. If we set act.sa_handler as SIG_IGN then the program simply keeps on looping as we are ignoring the SIG_INT signal.

In the POSIX base standard, a signal handler is an ordinary function that returns void and has one integer parameter. When the operating system delivers the signal, it sets this parameter to the number of the signal that was delivered. Most signal handlers ignore this value, but it is possible to have a single signal handler for many signals. The usefulness of signal handlers is limited by the inability to pass values to them. This capability has been added to the POSIX:RTS and POSIX:XSI Extensions, which can use the alternative sa_sigaction field of the struct sigaction structure to specify a handler.

If SA_SIGINFO is specified in sa_flags, then sa_sigaction (instead of sa_handler) specifies the signal-handling function for signum. This function receives the signal number as its first argument, a pointer to a siginfo_t as its second argument and a pointer to ucontext_t (cast to void *) as its third argument. The siginfo_t argument to sa_sigaction is a struct with the following elements:

siginfo_t {
   int      si_signo;    /* Signal number */
   int      si_errno;    /* An errno value */
   int      si_code;     /* Signal code */
   int      si_trapno;   /* Trap number that caused
                       hardware-generated signal
                       (unused on most architectures)*/
   pid_t    si_pid;      /* Sending process ID */
   uid_t    si_uid;      /* Real user ID of sending
                         process */
   int      si_status;   /* Exit value or signal */
   clock_t  si_utime;    /* User time consumed */
   clock_t  si_stime;    /* System time consumed */
   sigval_t si_value;    /* Signal value */
   int      si_int;      /* POSIX.1b signal */
   void    *si_ptr;      /* POSIX.1b signal */
   int      si_overrun;  /* Timer overrun count;
                         POSIX.1b timers */
   int      si_timerid;  /* Timer ID; POSIX.1b timers */
   void    *si_addr;     /* Memory location which
                         caused fault */
   int      si_band;     /* Band event */
   int      si_fd;       /* File descriptor */
}

NOTE: si_signo, si_errno and si_code are defined for all signals (si_errno is generally unused on Linux). The rest of the struct may be a union, so that one should only read the fields that are meaningful for the given signal.

static volatile sig_atomic_t doneflag = 0;

static void setdoneflag(int sig, siginfo_t *siginfo, void *context)
{
  printf ("Signo     -  %d\n",siginfo->si_signo);
  printf ("SigCode   -  %d\n",siginfo->si_code);
  doneflag = 1;
}

int main (int argc, char *argv[])
{
  struct sigaction act;

  memset (&act, '\0', sizeof(act));

  act.sa_sigaction = &setdoneflag;

  act.sa_flags = SA_SIGINFO;

  if (sigaction(SIGINT, &act, NULL) < 0) {
      perror ("sigaction");
      return 1;
  }

  while (!doneflag) {
      printf("press CTRL+C to kill the Loop\n");
      sleep(1);
  }

  printf("Program terminating ...\n");
  return 0;
}

Output:

$ ./a.out
press CTRL+C to kill the Loop
press CTRL+C to kill the Loop
^C
Signo     -  2
SigCode   -  128
Program terminating ...
$ 

Signals in Linux - Catching and Ignoring Signals - signal

The simplest way to change the action for a signal is to use the signal system call. You can specify a built-in action (such as to ignore the signal), or you can establish a handler.

#include <signal.h>

typedef void (*sighandler_t)(int);

sighandler_t signal(int signum, sighandler_t handler)

signal sets the disposition of the signal signum to handler, which is either SIG_IGN, SIG_DFL, or the address of a programmer-defined function (a "signal handler").

NOTE: The signals SIGKILL and SIGSTOP cannot be caught or ignored. The effects of signal system call in a multithreaded process are unspecified.

For more information checkout: man 2 signal

NOTE: The behavior of signal system call varies across different UNIX versions, and has also varied historically across different versions of Linux. Avoid its use: use sigaction system call instead. Check man 2 signal for detailed information about various portability issues.

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

static volatile sig_atomic_t doneflag = 0;

static void setdoneflag(int signo) {
    printf("\nIn SignalHandler - setdoneflag\n");
    doneflag = 1;
}

int main (void) {

    signal(SIGINT, setdoneflag);

    while (!doneflag) {
        printf("press CTRL+C to kill the Loop\n");
        sleep(1);
    }

    printf("Program terminating ...\n");
    return 0;
}

Output:

$ ./a.out
press CTRL+C to kill the Loop
press CTRL+C to kill the Loop
press CTRL+C to kill the Loop
^C
In SignalHandler - setdoneflag
Program terminating ...
$ 

How to: Listing all users in a Linux machine

TO list all the users who can access a Linux machine we have to access the /etc/passwd file, which stores information about all registered users of that machine. But it is not really so easy as told above since the file contains many other fields & machine trust accounts & inbuilt accounts.

We'll start by

cat /etc/passwd 

As we all know that by default all the users created will have their home directories in /home share so we'll modify our command a bit by using grep. Now it'll be

cat /etc/passwd | grep "/home"

Now we'll get all the user accounts which have their home share in /home.But the only output we need is the list of users & nothing else. So we'll modify our command again

cat /etc/passwd | grep "/home" |cut -d: -f1

Now what we have done is that we have piped the output of previous command to another variable "cut"

What we have done here is we have added cut -d: -f1
-d: means delimiter :
-f1 means display first field of line i.e. username.

So final command is

cat /etc/passwd | grep "/home" |cut -d: -f1

This works until all your users have their home share in /home. If you have defined their home share to some other destination. Modify the above command accordingly.

Signals in Linux - Generating Signals

Besides signals that are generated as a result of a hardware trap or interrupt, your program can explicitly send signals to itself or to another process.

The kill system call can be used to send any signal to any process group or process.

#include <sys/types.h>
#include <signal.h>

int kill(pid_t pid, int sig);

For more information checkout: man 2 kill

There are restrictions that prevent you from using kill to send signals to any random process. These are intended to prevent antisocial behavior such as arbitrarily killing off processes belonging to another user. In typical use, kill is used to pass signals between parent, child, and sibling processes, and in these situations you normally do have permission to send signals. The only common exception is when you run a setuid program in a child process; if the program changes its real UID as well as its effective UID, you may not have permission to send a signal. The su program does this.

A process or thread can send a signal to itself with the raise function. The raise function takes just one parameter, a signal number.

In a single-threaded program it is equivalent to kill(getpid(), sig). In a multithreaded program it is equivalent to pthread_kill(pthread_self(), sig). If the signal causes a handler to be called, raise will only return after the signal handler has returned.

#include <signal.h>

int raise(int sig);

For more information checkout: man 3 raise

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

static volatile sig_atomic_t doneflag = 10;

static void setdoneflag(int signo) {
    printf("\nIn SignalHandler - setdoneflag\n");
    doneflag=0;
}

int main (void) {

    signal(SIGINT, setdoneflag);

    while(doneflag--)
    {
        printf("In While loop - %d\n",doneflag);
        if(doneflag==5)
        raise(2);
        else
        sleep(1);
    }

    printf("Program terminating ...\n");
    return 0;
}