Synchronous Execution in Bash

Synchronous_bash

A more hands-on approach to answer the question – Why does ps o/p list the grep process after the pipe?

So what happens when we run commands in the bash shell using the pipe in between ? Are the processes launched after this synchronous or asynchrous?

TL; DR – they are synchrnous. Lets see how…

So when we launch things like “ps ajx | grep cat” – first the pipe is created, then the first process is launched (ps), then the second process is launched (grep) and then these two processes run to completion at the same time. One being the producer and the second one the consumer. After some brief amount of time the processes execute synchronously.

This is the final output we see:
This is how you want to trace it if you want to see everything in action:  

We first start with execev of bash, followed by a call to pipe and then 2 calls to clone, to create 2 children:

This is what the first child is doing:    

This is what the second child is doing:

So in effect we see that first pipe is created and then the two processes execute more or less synchronously.

 

Posted in bash, observations | Leave a comment

Where are the permission bits stored for a file in Linux ?

In short, in the inode, you can see the inode datastructure at the end.

__le16   i_mode;           /* File mode */ – It’s stored here. This is not a bit-field anymore.

Stat system call uses the following structure to retrieve:

unsigned int        st_mode; – This is where the permissions are stored. This is how you extract them, after you make the stat system call on the file:

References: How to implement ls in code, Linux Kernel and Glibc Code.

 

Posted in sysadmin, This works ! | Leave a comment

Tracing bash

Lets try to understand how echo list=$(ls) is executed exactly, we can see the explanation here, but you know proof of the pudding and all :-).

4844160895_33b8a673db_o

 

 

 

 

 

 

 

 

 

But simply running strace will not do ! We get a trace which is partial and we cannot see the shell in action. So we need to use a few tricks, and more tricks.

Let’s use the excellent systems utility – strace to dig deeper:

$ strace echo list=$(ls)

Here are running strace on the echo command, without using the -f option. The -f option will also trace the child processes as they are created by currently traced processes.

So the above output is quite simple and does not show everything in action, lets go ahead and try some tricks.

$ cat | strace -s 2000 -f bash > /dev/null

Once you run this line combination, you will be able to see how your input is read and then acted upon. Just put in the command you want to trace, and it will be piped to bash on which strace is attached.

$ { strace -s 2000 -f -p “$$” & sleep 1; echo list=$(ls) ; kill “%1″; } Here what we are doing is creating an anonymous(?) bash function with the curly braces “{}”. Inside this function, we are attaching strace to the current process – “$$” is the PID of the current process. We sleep for a second with “sleep 1” and then we go ahead and give the command we want to trace to our current bash process “echo list=$(ls)”. And finally we kill the strace which was job number 1 – %1 in the current shell.

$ strace -s 2000 -f bash -c ‘echo list=$(ls)’

Here we are attaching strace to a bash that we will spawn, and to that bash shell we will send the commands we want to trace. But again we miss the details we see in the above ways, though we see the bash shell being setup.

I am sure I am missing a few more way this can be done, please do let me know via comments.
In the next article, lets see how the bash exploit vectors work by using the above methods.

C

Photo Courtesy – Link

Posted in bash, LoVe ThiS ComManD !!, observations, sysadmin, This works !, ubuntu | Leave a comment

How to get started with Dtrace

Well, this is a fluff piece, I was looking for absolutely any excuse to use this PNG in my posts :-).

Dtrace Pony

So this is kind of a notes to self written while listening to the excellent Bryan Cantrill Google Tech Talk.

How to debug a slow system using dtrace –

List all the probes of dtrace

dtrace -l | wc -l

Trace all the system calls entry being done in real time – and output the execname which is doing it:

dtrace -n syscall:::entry'{trace(execname)}’

Aggregate all the exec names and count the number of system call entries they make

dtrace -n syscall:::entry'{@[execname] = count()}’ – Here we discover who is making a lot of sys calls – now here get the process name – like iTerm or Virtualbox

Call output every 5 seconds:

‘syscall:::entry { @[probefunc] = count() } tick-5sec { printa(@); cleara(@); }’

Now focus on the process and check what its doing exactly, this will tell us how many syscalls iTerm is calling and also print the pid of the process alongside it.
This is also useful when we want to aggregate many iTerms across the system.

dtrace -n syscall:::entry’/execname == “iTerm”/{@[probefunc,pid] = count()}’

Now we dig a little deeper into the process, that we have identified is interesting:

With this we take a backtrace whenever iTerm is calling read syscall and that will produce a backtrace of what iTerm was doing when it called read syscall.

dtrace -n syscall::read:entry’/execname == “iTerm”/{@[ustack()] = count()}’

Now here we see the app functions – from that you can select what is the most interesting and proceed further:

Now we will be using the pid provider which can instrument each function call inside of the app:

Here we are saying that for the pid 549 we should print the message in printf when we enter the function __CFRun…. This will also print the timestamp at that point.

dtrace -n pid549::__CFRunLoopRun:entry'{printf(“Called __CFRunLoopRun at %Y\n”, walltimestamp)}’ -q

Now we want to know what __CFR… is doing exactly – so we write this script:

Now with this we are instrumenting the whole process – iTerm

The o/p will show us what __CFR excatly did

#!/usr/sbin/dtrace -s

pid549::__CFRunLoopRun:entry
{
self->folow = 1;
}

pid549::__CFRunLoopRun:entry,
pid549::__CFRunLoopRun:return
/self->follow/
{}

pid549::__CFRunLoopRun:return
/self->follow/
{
self->follow = 0;
exit(0);
}

#pragma D option flowindent – this will print pretty

Now lets instrument the entire kernel:

What this will give us is – what exactly is __CFRun doing and will print the entire function entry and return points including the kernel points. It will tell us about the life of the entire function.

#!/usr/sbin/dtrace -s

#pragma D option flowindent

pid549::__CFRunLoopRun:entry
{
self->folow = 1;
}

fbt:::
/self->follow/
{}

pid549::__CFRunLoopRun:return
/self->follow/
{
self->follow = 0;
exit(0);
}

We can see the whole pre-syscall and post-syscall too ! This will be very descriptive.

C

Posted in Dtrace, LoVe ThiS ComManD !!, LoVe ThiS ShoRtCut !!, sysadmin, This works ! | Tagged | Leave a comment

Yet Another Interception Method (YAIM) – Part 1

So, lately I have been involved with porting Cisco vPath Protocol to our new stack. Apart from this I have also been dabbling with WCCP implementation in our stack. Both are Interception Protocols and have a set of common features. That got me thinking, what does it take to design a new (YAIM) ? In this series of articles I plan to design one based on OpenFlow Protocol Specifications, exploring the various options in process.

Pipes

Interception protocol or Interception is just one of the few ways to talk about this “traffic interception” method. Some other names given to this methodology are – “Interception”, “Content Routing”, “Service Insertion”, “Interposition” and “Traffic Redirection”. All of these more or less mean the same, in the context of these series of articles.

We want to intercept interesting traffic, like HTTP and get it over to our content engines to do interesting things to it. And one of the important needs of this Interception is that it should be transparent to the clients and servers and if possible other elements of your network topology.

So to start with, lets list down the basic elements of an Interception Protocol, in no particular order:

  • Host Management
  • Traffic Management
  • Traffic Redirection
  • Traffic Return from Content Engines
  • Traffic Egress from Content Engines
  • Availability / Heartbeat Management of Content Engines and Content Routers
  • Cluster Management
  • Application Aware Protocol Redirection
  • Flow Aware Protocol Redirection
  • Support for Traffic Reinjection
  • Security
  • Support for Configuration of the Content Engine/s
  • Support for Configuration of the Content Router/s
  • Support for Chaining of more than one Content Engines
  • Support for Multiple Protocols
  • Support for Registration / Consensus management of Content Engines and Content Routers
  • Support for Assignment, Load Distribution of Content to Content Engines by Content Routers
  • Support for Advanced Features, which might include Deep Packet Inspection etc.

In the next set of articles we will explore how OpenFlow Protocol can provide us with each of the above.

Comments/Suggestions most welcome !

C

Photo Credit: Pipes by  Mathias Liebing License.

Posted in Uncategorized | Leave a comment

chmod -x /bin/chmod Doh !

Ok so, you did it ! chmod -x /bin/chmod

There are many ways to solve this but – 2 ways are really clever and here is an strace to see how one of them works. 

The first one is to use the execute permission bit set in a diffrent file and use that to set the permission bit in the chmod binary. Here we use autoconf – which is a script, but anything else can be used too. 

cp -p /usr/bin/autoconf /tmp/scratch
cat /bin/chmod > /tmp/scratch
cp -p /tmp/scratch /bin/chmod

In the second one we use the linux linker/loader to load our file and execute it, bypassing the exec permission check. 

Lets see how this one works:

Here is the normal execution:

root@Boss-09:47:49/~# strace chmod +x exec.bin
execve("/bin/chmod", ["chmod", "+x", "exec.bin"], [/* 22 vars */]) = 0
brk(0)                                  = 0x1703000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f865952a000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=58597, ...}) = 0
mmap(NULL, 58597, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f865951b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1845024, ...}) = 0
mmap(NULL, 3953344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f8658f44000
mprotect(0x7f8659100000, 2093056, PROT_NONE) = 0
mmap(0x7f86592ff000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bb000) = 0x7f86592ff000
mmap(0x7f8659305000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f8659305000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f865951a000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f8659518000
arch_prctl(ARCH_SET_FS, 0x7f8659518740) = 0
mprotect(0x7f86592ff000, 16384, PROT_READ) = 0
mprotect(0x60c000, 4096, PROT_READ)     = 0
mprotect(0x7f865952c000, 4096, PROT_READ) = 0
munmap(0x7f865951b000, 58597)           = 0
brk(0)                                  = 0x1703000
brk(0x1724000)                          = 0x1724000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2919792, ...}) = 0
mmap(NULL, 2919792, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f8658c7b000
close(3)                                = 0
umask(0)                                = 022
stat("exec.bin", {st_mode=S_IFREG|0755, st_size=0, ...}) = 0
fchmodat(AT_FDCWD, "exec.bin", 0755)    = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Here is the execution by using the linux linker/loader – /lib/x86_64-linux-gnu/ld-2.19.so (in my system):

root@Boss-09:54:47/~# strace /lib/x86_64-linux-gnu/ld-2.19.so ./chmod-x +x exec.bin
execve("/lib/x86_64-linux-gnu/ld-2.19.so", ["/lib/x86_64-linux-gnu/ld-2.19.so", "./chmod-x", "+x", "exec.bin"], [/* 22 vars */]) = 0
brk(0)                                  = 0x7f9a719ef000
open("./chmod-x", O_RDONLY|O_CLOEXEC)   = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\2\0>\0\1\0\0\0\\&@\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=56032, ...}) = 0
getcwd("/root", 128)                    = 6
mmap(0x400000, 53248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400000
mmap(0x60c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xc000) = 0x60c000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9a6ffbc000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=58597, ...}) = 0
mmap(NULL, 58597, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f9a6ffad000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1845024, ...}) = 0
mmap(NULL, 3953344, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f9a6fbe7000
mprotect(0x7f9a6fda3000, 2093056, PROT_NONE) = 0
mmap(0x7f9a6ffa2000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bb000) = 0x7f9a6ffa2000
mmap(0x7f9a6ffa8000, 17088, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f9a6ffa8000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9a6fbe6000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9a6fbe4000
arch_prctl(ARCH_SET_FS, 0x7f9a6fbe4740) = 0
mprotect(0x7f9a6ffa2000, 16384, PROT_READ) = 0
mprotect(0x60c000, 4096, PROT_READ)     = 0
mprotect(0x7f9a701e0000, 4096, PROT_READ) = 0
munmap(0x7f9a6ffad000, 58597)           = 0
brk(0)                                  = 0x7f9a719ef000
brk(0x7f9a71a10000)                     = 0x7f9a71a10000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2919792, ...}) = 0
mmap(NULL, 2919792, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f9a6f91b000
close(3)                                = 0
umask(0)                                = 022
stat("exec.bin", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
fchmodat(AT_FDCWD, "exec.bin", 0755)    = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++ 

Source:

Excellent mail thread discussion

Presentation all about this by 

Posted in LoVe ThiS ComManD !!, sysadmin, This works ! | Leave a comment

From network card to user-space in (almost) no time

Getting packets into userspace

Recently I was trying to get the vPath library to work in user-space.  What I wanted was to try it out before working on the production code, so I decided that I wanted it as a simple user-space program. The thing with exercising the library as it would be worked out in the real world code is that we need to get packets to it, make it do stuff and then return the packets back.

There are many ways to get packets into user-space, we have netmap, Intel DPDK, TUN/TAP interface, etc. (more can be found here). My first choice was netmap, the challenge with that is to get just a few packets into my program it sounded like lot of work. So I have put a pin in it to come back to it later.

Other way to do it would be to use tools like TCPreplay, wireplay, hping3, Scapy, Unix Raw Sockets, etc. in which we can craft packets like we want and send them to the target. But that was a no go too. Because even if I did have access to previous versions of TCP traffic, it’s nothing like the real world traffic. After a significant overhead the rewards are but a few in this case. Though these tools are like gold when it comes to recreating customer cases, for my current scenario, the overhead was too much for too little benefit.

To get a quick and dirty 5 min solution, I turned to TUN/TAP interfaces. These are neat little buggers, fitted the purpose nicely. You can read more about how to program them here.

As is the case in real world, I solved one problem just to get to the next one :-). Now the thing is even if we have TUN/TAP we need the real world traffic to come to it without any modifications. For the purposes of my exercise I needed the L2 header to be intact. That’s when after searching around for a solution, playing around with linux bridge and Openvswitch, I found out this neat little utility – netsniff-ng. And that was the answer to (almost) all of my problems.

Now the one feature of netsniff-ng I ended up using was its “—mmap”, and as per the authors it is “Useful to have raw access to network packet data in user space”.

The netsniff-ng website states – “netsniff-ng, a fast zero-copy analyzer, pcap capturing and replaying tool”, the only part I have used is the fast zero-copy analyzer.

This is the way I hooked up the tap0 to eth1 –

ifconfig tap0 up  ; /root/netsniff-ng/netsniff-ng/netsniff-ng –in eth1 –out tap0 –mmap –prio-high

Finally, the problem I did not solve –

I just wanted to exercise my state machine by feeding it traffic, so I was not really worried about sending the traffic back, though I did manage to send it back this way –

ifconfig tap0 up  ; ( /root/netsniff-ng/netsniff-ng/netsniff-ng –in eth1 –out tap0 –mmap –prio-high & /root/netsniff-ng/netsniff-ng/netsniff-ng –in tap0 –out eth1 –mmap –prio-high )

But I know it can be done in a better way. Thats slated for another day, another venture.

C

Image credit: Thomas HawkLicenseImage Link

 

Posted in Generic X86 Networking, Network Virtualization, Programming, This works !, Userspace Networking | Leave a comment

Finding innovation

2 tweets got me thinking:

&

In today’s world when you can really build a prototype and get running all on your own, is it really hard to innovate ? Are really really good ideas very very hard to come by ?

I think not really, there is room for innovation, still and always will be. There will not be a time ever when a person can claim – “All the innovation is done”.

We will have innovations, where solutions from one dimension are brought into the other. Like the MakerBot industries got the 3D printing solutions from the enterprise and really pricy dimension to the dimension of generally available technology.

Or innovation like Navisens which will reinvent something like tracking a moving body by using all the normal sensors which are available in the common smartphone, location of device placement & some clever mathematics. They are moving solutions from the dimension of thought to the dimension of productization.

Or there will be innovation like that of Google glass, where the power of millions of minds will come to the aid of a single one.

Or something like Trailerpop, where somethings which were found in the dimension of competitive games and related rewards is being brought into the dimension of content discovery.

Innovation will always be around, it was before and it will be after. But what is lacking is the will to build, will to create and will to make ideas turn into products. It was this will which gave necessary power to the products to be drawn from one dimension to another.

And at the end – persevere and try try try :-) success will be yours !

C

Posted in musings, observations, Thoughts | Leave a comment

Truth about programing – Many ways of doing things – each one of them is right !

Was reading the excellent book – Assembly Language Step-by-Step: Programming with Linux by Jeff Duntemann, in that he quotes from Rudyard Kipling’s poem “In the Neolithic Age“.

Seems really applicable to coding with any language:

But my Totem saw the shame; from his ridgepole-shrine he came,
And he told me in a vision of the night: —
“There are nine and sixty ways of constructing tribal lays,
“And every single one of them is right!”

Yes many ways to do things and in this case every single one of them is right !

C

Posted in GoodReAd, musings | Leave a comment

ROCKBOT @Launch 2013

What was #launch2013 rocking to – http://rockbot.com/venues/main-hall-launch-conference – Awesome selection !

C

Posted in observations | Leave a comment