El Demonio Negro

Una curiosidad que permitirá conocer los entresijos de un ejecutable en LINUX.
Lectura recomendable, aunque sea para una lectura rápida y a saltos. Cuenta cómo va optimizando el código para pasar de 3998 bytes a 45 bytes.

Referencias y enlaces

A Whirlwind Tutorial on Creating Really Teensy ELF Executables for LINUX

Introduction

In this post I’m going to try to connect two processes in different machines through DBus. The method is a little bit complex, so be patient if you try.
Also is to advert that this has been the result of 3 days of tests (reference1). So maybe this method may be improved with time and use reference2.

Tools (The actors)

• dbus
• gabriel
  • socat
  • libssh
• SSH
• your apps

Debian official packages are dbus libssh-2 socat
gabriel is not part of Debian yet (but I’ve build one for myself)

Knowledge (Actors curriculum)

In this section I will describe the basics about the tools we are going to use.

DBus. Extracted from DBus page:

D-Bus is a message bus system, a simple way for applications to talk to one another. In addition to interprocess communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a “single instance” application or daemon, and to launch applications and daemons on demand when their services are needed.

D-Bus supplies both a system daemon (for events such as “new hardware device added” or “printer queue changed”) and a per-user-login-session daemon (for general IPC needs among user applications). Also, the message bus is built on top of a general one-to-one message passing framework, which can be used by any two apps to communicate directly (without going through the message bus daemon). Currently the communicating applications are on one computer, or through unencrypted TCP/IP suitable for use behind a firewall with shared NFS home directories.

Gabriel is a simple utility to enable D-Bus clients to connect to a D-Bus daemon running on a remote machine, through SSH.
This is the main piece of this puzzle. If you are interested in understanding how it works you should take a look at socat and libssh. As I’ve had to take a look at code, and make some modifications, you should read it as a punishment.

Extracted from socat man page:

socat - Multipurpose relay (SOcket CAT)
socat is a command line based utility that establishes two bidirectional byte streams and transfers data between them. Because the streams can be constructed from a large set of different types of data sinks and sources (see address types), and because lots of address options may be applied to the streams, socat can be used for many different purposes. It might be one of the tools that one ‘has already needed´.

Libssh. Extracted from libssh page:

The SSH library was designed to be used by programmers needing a working SSH implementation by the mean of a library. The complete control of the client is made by the programmer. With libssh, you can remotely execute programs, transfer files, use a secure and transparent tunnel for your remote programs. With its Secure FTP implementation, you can play with remote files easily, without third-party programs others than libcrypto (from openssl).

You should know about SSH and about your application.

Architecture

Local host will run gabriel and your application.
Remove host will need a running SSH server, a running dbus server and will need socat installed and ready to use.
We need to run gabriel, that will act as a server that will connect our host to the remote host through SSH. After that gabriel will use this SSH connection to intercommunicate our local application with remote DBus applications by using socat.

Remote DBus communication

Howto (Main action)

At the moment I’ve only achieved to connect a process using session-bus, I’m still testing until I get connection through system-bus which was my initial purpose.
After reading next information, you will be able to connect using session bus and system bus.

As I commented somewhere else, I’ve made some modifications on gabriel code. I needed some common parameters as SSH port (my virtualbox testing environment ), better help explanations or add a verbose output.
Gabriel establish a connection with the remote SSH and by socat commands it communicates with the remote DBus “environment”. You should administrate SSH parameters and Dbus parameters to gabriel.

user@machine:~/svn/gabriel/gabriel$ src/gabriel –help
Usage:
gabriel [OPTION...] - Gabriel
Help Options:
-?, –help Show help options
Application Options:
-h, –host=HOSTNAME Hostname or IP of the remote host
-p, –port-ssh=PORT-SSH SSH port on the remote host
-u, –username=USERNAME SSH username on the remote host
-w, –password=PASSWORD SSH password on the remote host
-m, –method=DBUS_TRANSPORT_METHOD The D-Bus transport method to use (TCP, UNIX, abstract-UNIX)
-b, –bind=HOSTNAME The bind-address to listen for D-Bus client connections on
-d, –bus-address=BUS_ADDRESS The DBus session bus address of the remote D-Bus daemon
-t, –port-tcp=PORT-TCP The TCP port to listen for DBus client connections on
-v, –verbose=VERBOSE Set verbosity level (3, 2, 1, 0, -1)=(packet,protocol,functions,important,none)

We have to put special attention to -d, –bus-address=BUS_ADDRESS because this info must be gotten from the REMOTE machine.
That address is the one used by processes to communicate through DBUS. It’s something “internal” and automatically done when you use DBus API/library. I’m going to show you where to get it.

DBUS_SESSION_BUS_ADDRESS, DBUS_SYSTEM_BUS_ADDRESS, DBUS_SYSTEM_BUS_DEFAULT_ADDRESS

Again, this info should be gotten from REMOTE machine.
At the moment I don’t know any nice command where to get this info.
We have two main options of DBus buses. System and Session (more info in DBus page).
If you need SESSION bus address, you can choose what it better fits you:

• You can can get it from process environment
• You can stole it from any other process suspicious from being involved in DBus activities…
• You can create your own dbus-daemon (which, actually, I don’t know if it uses it’s own BUS_ADDRESS)

If you need SYSTEM bus address, you can choose what it better fits you:

• You can can get it from process environment. If it’s not defined, take a look at /etc/dbus-1/system.conf where you should locate a string like <listen>unix:path=/var/run/dbus/system_bus_socket</listen>
• You can stole it from any other process suspicious from being involved in DBus activities…

Examples:

// C++ code
#include <stdio.h>
#include <stdlib.h>
char * env;
// env = getenv (”DBUS_SESSION_BUS_ADDRESS”);
env = getenv (”DBUS_SYSTEM_BUS_ADDRESS”);
if (env!=NULL){
cout << env << endl;
}

user@REMOTE-machine:~ $ grep -z BUS_ADDRESS /proc/2047/environ
HALD_RUNNER_DBUS_ADDRESS=unix:abstract=/var/run/hald/dbus-bonbZtoykd,guid=8b5aff565de0dc7c467fef414832dd98

This command gives you a dbus-daemon in your session with the one you can contact.

user@REMOTE-machine:~ $ dbus-daemon –session –print-address
unix:abstract=/tmp/dbus-j2npbTXDwD,guid=d69f66d43e354de5733e4a1a48335bd6

user@REMOTE-machine:~ $ grep unix /etc/dbus-1/system.conf
<listen>unix:path=/var/run/dbus/system_bus_socket</listen>

Howto (Main action): Back to local host

Those ugly unix:stri:ngs/asdkaj/numbers we have seen is what we need for -d, –bus-address=BUS_ADDRESS.
See a session example:

user@machine:~/svn/gabriel/gabriel$ src/gabriel -h localhost -p 2222 -d unix:abstract=/tmp/dbus-j2npbTXDwD,guid=d69f66d43e354de5733e4a1a48335bd6
Listening to D-Bus clients on: “unix:abstract=/tmp/gabriel”
bla ble blu bla

See a system example:

user@machine:~/svn/gabriel/gabriel$ src/gabriel -h localhost -p 2222 -d unix:path=/var/run/dbus/system_bus_socket
Listening to D-Bus clients on: “unix:abstract=/tmp/gabriel”
bla ble blu bla

The moment we have or gabriel server running we (may have nothing) need to set DBUS_XXX_BUS_ADDRESS. Many apps would use, or have, this environment variable to connect to a DBus instance and intercommunicate with other process.
This is is easy, DBUS_XXX_BUS_ADDRESS should be the address gabriel shows few instants after being launched.
When we have defined this environment variable (in command line) we can execute our app, and it will happily communicate with the remote DBus world.
Example:

user@machine:~/svn/dbusmm/trunk/examples/glib$ export DBUS_SESSION_BUS_ADDRESS=”unix:abstract=/tmp/gabriel”
user@machine:~/svn/dbusmm/trunk/examples/glib$ ./dbus-browser

dbus-browser is a program that uses a session bus.

Curiosity: DBus protocol messages interchanged

Modifying a couple of lines in gabriel can let you see DBus raw protocol messages. It’s a didactic info.
If you enable verbose code at least at level 2, you will get raw DBus protocol messages.

My modifications and hacks

Code will be publish under GLKM project page.

Links and references

• dbus site
• gabriel site
• socat site
• libssh site
• reference 1. (informational note, it had implied jumping into gabriel, libssh, and dbus code and testing with a virtualbox machine)
• reference 2. (personal note, take a look at “Securing traffic between two socat instances using SSL” article in socat page)

This is not an original article (see references), it’s just an archived one “por si las moscas”.

Introduction

By default, the network connection in Virtualbox is set to NAT, that is every packet coming from the Guest machine is modified so that it seems as it has come from the Host machine. In this way it’s easy for the Guest machine to connect to all the rest of the network (the Internet included) but nobody can start a connection with the Guest Machine since it’s hidden behind the Host one.

So, if you want to use a server service in your Guest machine (i.e. Apache or SSH) you have two choices:

1. pass to Virtualbox Host network connection;
2. make Virtualbox forward all the packets arriving to a certain port of the Host machine.

This article will describe how to do the latter, in particular in the case of the SSH server. This is an interesting case because it allows you to simulate very well a quite common condition: connecting to a remote LINUX headless machine.

We have a guest machine with a running SSH server which accepts connections on the TCP port 22.
Our goal is to make any packet arriving at a given TCP port (i.e. 2222) of the host machine, to be forwarded to the TCP port 22 of the guest machine.
Fortunately, there is Virtualbox command which permits to do it almost instantly: VBoxManage.

Configuration needed

In our case we will use Debian as the guest machine name (quote in case guest machine name contains spaces), here are the commands that you have to type in the host machine (real one) console as your user (if you use another user it will try to look for its virtual machines):

user@machine $ VBoxManage list vms
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
Name: debian
Guest OS: LINUX 2.6
UUID: 814e25f4-451e-4582-8d34-71a1cd437cdd
Config file: /mnt/extra/virtualizacion/virtualbox/debian/debian.xml
Memory size: 195MB
[...]
user@machine $ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort” 2222
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
user@machine $ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort” 22
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
user@machine$ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol” TCP
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.

HostPort,GuestPort,Protocol are just string examples and can be changed without any consequence. Just remember your choices if you want to remove them later.
I guess you should be careful with pcnet substring, as maybe it should have something to see with your selected network card type.

Testing

Once you have typed the above commands, you need to close the guest machine (a reboot won’t be sufficient), restart it and then connect via SSH with:

anyuser@machine$ SSH -l >user< -p 2222 localhost

Replace localhost with the host machine IP address if you are connecting from another computer.

By the way, you can check which customizations have been already set for your Guest Machine with VBoxManage by typing:

user@machine$ VBoxManage getextradata debian enumerate
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
Key: GUI/LastWindowPostion, Value: 0,6,1028,820
Key: GUI/Fullscreen, Value: off
Key: GUI/AutoresizeGuest, Value: off
Key: GUI/LastCloseAction, Value: powerOff
Key: GUI/SaveMountedAtRuntime, Value: yes
Key: GUI/Seamless, Value: off
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort, Value: 2222
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort, Value: 22
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol, Value: TCP

or remove one, for example “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort”, by setting it without any value:

user@machine$ VBoxManage setextradata “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort”

Links and references

• an article about VirtualBox on doc.ubuntu-fr.org (which offers a slightly different solution)
• Howto access via SSH a Virtualbox guest machine.

¿Cuándo?

En los días 8 y 9 del mes de Mayo se celebrará la cuarta edición de la WhyFLOSS Conference, con entrada libre y gratuita.

Con un importante apoyo de la Escuela de Informática de la Universidad Politécnica de Madrid, Campus Sur se presentarán conferencias variadas entorno a las tecnologías abiertas de IT.

Estarán colaborando en la IV edición del evento compañí­as como SUN, Red Hat, OpenBravo, Andago, SIE, Liferay, Opentia, Monolabs, Accenture, Avanzada7 y universidades como la UPM y la URJC, así­ como comunidades de proyectos como LibreSoft, OpenSolaris y FFII.

¿Dónde?

El evento se realizará en la Escuela de Informática de la Universidad Politécnica de Madrid. Se encuentra ubicada en el Campus Sur de la UPM en la carretera de Valencia Km. 7 en la Ciudad de Madrid (España).
Información de localización de la Whyfloss Madrid 2008

Calendario

DIA 1

9:30
Inauguración WHYFLOSS Conference 08.
Alejandro Sánchez Acosta, Neurowork

10:00
Open-Cities: el reto de la administración electrónica
Guillermo Pastor, Ándago Ingeniería S.L.

11:00
VII Programa Marco en la UE: FLOSS Include y FLOSS Metrics
Jesús Gonzalez Barahona, LibreSoft

12:00
Modelos de negocio basados en Asterisk (la plataforma de VoIP basada en Software Libre)
Juan Ignacio Cabrera, Avanzada 7

13:00
Clustering Computacional en CSIC
Raul Diaz Medina, Sistemas Informáticos Europeos S.L.

14:00
Descanso para comer

16:00
La implicación de la FFII en los estándares abiertos en Europa
Alberto Barrionuevo, Presidente de Foundation for a Free Information Infrastructure (FFII) / OPENTIA, S.L.

17:00
Caso de exito OpenSolaris en Accenture
David Galan Ortiz, Accenture Outsourcing

18:00
¿Es viable el software Open Source en la Industria? El caso de Red Hat LINUX y JBoss
Jesús González de Buitrago, Red-Hat

DIA 2

10:00
Evolución e influencia del Software Libre en los 10 últimos años
Juantomás García, Monolabs

11:00
Liferay Enterprise Portal: The project, the product, the community and how to extend it
Alvaro del Castillo San Félix, Liferay Inc.

12:00
Openbravo: las claves del éxito del desarrollo en las aplicaciones en software libre
Representative, OpenBravo Inc

13:00
Rocks: Distribucion para clusters computacionales
Jesús Espino García, Sistemas Informáticos Europeos

14:00
Descanso para comer

16:00
Seguridad en OpenSolaris
Victor M. Fernandez, SIA / OpenSolaris Hispano

17:00
Django: Framework MVC en Python
Jesús Espino García, Sistemas Informáticos Europeos

Notas personales (idem a la pasada edición):

• He de decir que conozco al organizador principal.
• Yo voy

Referencias y enlaces

• Puedes registrarte vía web
• Si desea colaborar en la difusión del evento puede seguir este enlace.
• Puede consultar el programa definitivo.

Hello fellows.
My latest adventure was about moving a project (unmaintained) from its public subversion repository which of course I was not admin (which means no admin access) over to my own server.

Usually, to do this you’d dump the whole thing with svnadmin into one file (svnadmin dump > file_dump) and load it again at the new location (svnadmin load file_dump). After searching for any similar command without admin access into documentation I figured that something similar might exist for plain svn to svn migration.
I founded a ruby version and a python. I chose python one because it’s recent, it’s python and it’s under google code site.
There was other cause, the ruby version went through all revisions, from 1 till infinity, although many of them didn’t have any change. In the other hand, python version uses subversion logs and made a efficient use of it.

The real action

Create a repository in my server was easy:

root@aristoteles:/var/lib/svn$ rm -rf MY-PROJECT-PATH
root@aristoteles:/var/lib/svn$ svnadmin create MY-PROJECT-PATH
root@aristoteles:/var/lib/svn$ chgrp svn-user MY-PROJECT-PATH/ -R
root@aristoteles:/var/lib/svn$ chmod g+ws MY-PROJECT-PATH/ -R
root@aristoteles:/var/lib/svn$ chmod o-rx MY-PROJECT-PATH/ -R

After solving the problems I talk about, duplicating the repo was not so difficult.

user@othermachine:~$ mkdir tmp
user@othermachine:~$ cd tmp
user@othermachine:~$ svn co URL-DEST-MY-PROJECT-PATH
user@othermachine:~$ python svn2svn-0.1.1.py -r 7382 URL-ORIG URL-DEST-MY-PROJECT-PATH

This process took a while but the new directory was now ready for action.

Be careful with your subversion config files. It may cause conflicts with file in repositories.
As an example, I had some troubles because I use to ignore files like Makefile.in and so. The project I was trying to import did have files like that and svn2svn did bring svn errors because those files were not under revision. When I realized I modified my subversion config without that rules for a while.

References and links

• svn2svn

A simple command line tool called mpy-svn-stats give us nice graphics using svn XML output.

You can see an example GLKM project statistics

It’s quite easy to use, see:

user@machine:/var/www/svn-stats$ mpy-svn-stats -o ./ svn+ssh://localhost/var/lib/svn/glkm/
Will generate PIL graphs.
getting data
running command: “svn -v –XML log svn+ssh://localhost/var/lib/svn/glkm/”
[...ssh stuff...]
done
parsing data
done
calculating stats
done
writing data
done
Have 13 stats objects, 13 of them are wanted.
user@machine:/var/www/svn-stats$ ll
total 48
-rw-r–r– 1 user user 10379 2008-02-06 17:06 changed_paths_multi_author_graph.png
-rw-r–r– 1 user user 10268 2008-02-06 17:06 commits_group_multi_author_graph.png
-rw-r–r– 1 user user 11762 2008-02-06 17:06 index.html
-rw-r–r– 1 user user 10265 2008-02-06 17:06 log_message_length_group_multi_author_graph.png
user@machine:/var/www/svn-stats$

References and links

• project web page mpy-svn-stats

As I wrote some time ago for spanish readers, I was having HTML 404 errors in my logs (see awstats statistics before January 2008) which I wanted to solve.

My awstats collects web statistics from eldemonionegro.{com,org,es}/wordpress and eldemonionegro.{com,org,es}/gallery2. The problem should be solved in two different ways, as wordpress and gallery2 has two different approaches when rewriting URL request.
Problems in URL coming from wordpress were due to changes I did in the way that categories and some other URL staff were generated, so everything was rightly delimited.
In the other hand, problems with gallery2 arise from who knows where. With gallery2, HTLM 404 errors come from bad formed URLs, they are errors properly generated, but, those errors, wherever they come, needed to be transformed in proper URLs and as I don’t want to lose visitors, I wanted to move those, let’s say, “false URL” to good ones, so visitors are not loosed because of an external stupid engine. The point is that I solved those errors, but now others has appeared. “Stupids” are bringing me loads of fun ;).

How wordpress works is that it uses an internal rewriting system, with an external simple .htaccess in the root wordpress directory (at least in Debian with Apache2). If you want, you can see those internal rules using a plugin called internal rewrite viewer, it’s interesting.
Over this period you could also need some web server debug output activated (Apache2 in my case). Needed rules will be showed next, but use them carefully. As you should know, debuging is a terribly-horrible-madness performance killer.

# My own custom rules
<ifmodule mod_rewrite.c>
RewriteEngine On
RewriteBase /wordpress/
RewriteCond %{REQUEST_URI} /wordpress/archivos/tag/(.+?)/?$
RewriteRule . /wordpress/etiquetas/%1 [R=301,L]
RewriteCond %{REQUEST_URI} /wordpress/archivos/category/(.+?)/?$
RewriteRule . /wordpress/tema/%1 [R=301,L]
</ifmodule>
# End Custom Rules
# Rules automatically generated by wordpress
# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /wordpress/
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /wordpress/index.php [L]
</IfModule>
# END WordPress

Be careful of %{REQUEST_URI} instead of %{THE_REQUEST}

# In Debian my wordpress conf is saved into
# /etc/apache2/sites-enabled/wordpress.conf
<IfModule mod_rewrite.c>
RewriteLog “/var/log/apache2/rewrite.log”
# RewriteLogLevel 5
</IfModule>
…

If anyone’s interested in gallery2, I could show my suboptimal rules

En esta entrada sólo me limito a escribir las principales funcionalidades/objetivos que ya pensé de dotar a mi PFC. Son un poco datos que debí haber publicado hace ya algún tiempo (para variar).
A priori no tengo porque implementarlo todo para poder terminar, ya que como se puede observar, terminan siendo muchas cosas.

Este futuro monitor gráfico de procesos del kernel LINUX será útil para depurar procesos tanto de la propia máquina como de una máquina remota.
La información monitorizada corresponde a las estructuras internas del sistema operativo, más concretamente, del sistema de ficheros con respecto a los procesos.

• Orientado a una interfaz gráfica.
• Monitorización en red.
• Monitorización de procesos. Específicamente atributos de ficheros Reiser. Por supuesto la arquitectura va encaminada a extensiones que permitan ampliar las estructuras a monitorizar
• Visualizar/Monitorizar grupos de procesos simultáneamente.
• Estadísticas del uso de ficheros (opcional)
• Tipos de monitorización: Tiempo real o asíncrona. En tiempo real se toman datos cada intervalo X de tiempo de forma constante. En el caso de monitorización asíncrona se toman datos en los instantes que se indiquen.
• Integración con DBUS/HAL. En la medida de lo posible quiero llevar la comunicación entre la interfaz gráfica y el módulo del kernel por DBUS. A ver como sale.

No cabe duda que no quiero perder de vista al dtrace para LINUX, systemtap. Systemtap realiza labores similares al fin y al cabo.
Para quien no conozca estas herramientas en los siguiente párrafos hago un breve resumen.
Dtrace y systemtap son muy similares, sólo que una está ligada a Solaris y la otra a LINUX. Como en mi caso, monitorizar el núcleo del sistema operativo te cierne (un poco por definición) a ser una herramienta monoplataforma. Ambas proveen unos “lenguajes” propios que permiten extraer/monitorizar variables del sistema operativo.
Con dicha información se pueden realizar estadísticas y depuración como objetivo primordial.