strace - diagnostic, debugging and reverse engineering tool

Many times we come across hopeless situations where a program when compiled and installed in GNU/Linux just fails to run. Then we have to trace the output of the misbehaving program. But tracing the output of a program throws up a lot of data and it is a daunting task to go through volumes of data. Still there are cases where we are not fruitful in pin pointing the cause of error.

In this situation strace also known as system-call tracer comes for rescue. It is a debugging tool that monitors the system calls used by a program and all the signals it receives.

System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs. And programmers will find that since system calls and signals are events that happen at the user/kernel interface, a close examination of this boundary is very useful for bug isolation, sanity checking and attempting to capture race conditions.

A system call is the most common way programs communicate with the kernel. System calls include reading and writing data, opening and closing files and all kinds of network communication. Under Linux, a system call is done by calling a special interrupt with the number of the system call and its parameters stored in the CPU's registers.

Using strace is quite simple. There are two ways to let strace monitor a program.

FIRST METHOD
Start a program using strace command as shown below which prints the list of system calls made by the program.

strace program-name

For example let us trace uname command.

$strace uname
execve("/bin/uname", ["uname"], [/* 23 vars */]) = 0
brk(0)                                  = 0x871d000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f30000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=59428, ...}) = 0
mmap2(NULL, 59428, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7f21000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\220\272"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1577052, ...}) = 0
mmap2(0x45b93000, 1295780, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x45b93000
mmap2(0x45cca000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x137) = 0x45cca000
mmap2(0x45ccd000, 9636, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x45ccd000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f20000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb7f206c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x45cca000, 8192, PROT_READ)   = 0
mprotect(0x451dd000, 4096, PROT_READ)   = 0
munmap(0xb7f21000, 59428)               = 0
brk(0)                                  = 0x871d000
brk(0x873e000)                          = 0x873e000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=55511552, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7d20000
close(3)                                = 0
uname({sys="Linux", node="localhost.localdomain", ...}) = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f2f000
write(1, "Linux\n", 6Linux
)                  = 6
close(1)                                = 0
munmap(0xb7f2f000, 4096)                = 0
exit_group(0)                           = ?
Process 7477 detached

Even though output from strace looks very complicated, but this is only due to many system calls made when loading shared libraries. However, once we have found which system calls are the important ones (mainly open, read, write and the like), the results will look fairly intuitive to us.

SECOND METHOD

We can use the -p flag to attach to a running process and debug a running process. Let us try out on the sample code shown below.

#include <stdio.h>
#include <unistd.h>

int main()
{
   sleep(20);
   return 0;
}

We will compile the above code and run it as a background process. Then using its process id and -p flag we will run strace on it.

$ gcc main.c

$ ./a.out &
[1] 7609

$ strace -p 7609
Process 7609 attached - interrupt to quit
restart_syscall(<... resuming interrupted call ...>) = 0
exit_group(0)                           = ?
Process 7609 detached
[1]+  Done                    ./a.out

Using -p flag we even debug daemon processes. In contrast to a debugger, strace does not need a program's source code to produce human-readable output.
But sometimes strace produces great amount of data which creates problems in scrolling through huge output. In this case we can use -o option as shown below.

$ strace -o uname_output uname

If you use the -t option, then strace will prefix each line of the trace with the time of day. We can even specify the system call functions to trace using the -e option. For example, to trace only open() and close() function system calls use the following command:

$ strace -o uname_output -e trace=open,close uname

References:
1. OS Review

Static Functions in C

By default all functions are implicitly declared as extern, which means they're visible across translation units. But when we use static it restricts visibility of the function to the translation unit in which it's defined. So we can say

Functions that are visible only to other functions in the same file are known as static functions.

Let use try out some code about static functions.
main.c

#include "header.h"

int main()
{
hello();
return 0;
}

func.c

#include "header.h"

void hello()
{
printf("HELLO WORLD\n");
}

header.h

#include <stdio.h>

static void hello();

If we compile above code it fails as shown below

$gcc main.c func.c
header.h:4: warning: "hello" used but never defined
/tmp/ccaHx5Ic.o: In function `main':
main.c:(.text+0x12): undefined reference to `hello'
collect2: ld returned 1 exit status

It fails in Linking since function hello() is declared as static and its definition is accessible only within func.c file but not for main.c file. All the functions within func.c can access hello() function but not by functions outside func.c file.

Using this concept we can restrict others from accessing the internal functions which we want to hide from outside world. Now we don't need to create private header files for internal functions.

Note:
For some reason, static has different meanings in in different contexts.

1. When specified on a function declaration, it makes the function local to the file.
2. When specified with a variable inside a function, it allows the variable to retain its value between calls to the function. See static variables.

It seems a little strange that the same keyword has such different meanings...