Howto to intercommunicate processes in different(remote) machines through DBus

Introduction

In this post I’m going to try to connect two processes in different machines through DBus. The method is a little bit complex, so be patient if you try.
Also is to advert that this has been the result of 3 days of tests (reference1). So maybe this method may be improved with time and use reference2.

Tools (The actors)

  • dbus
  • gabriel
    • socat
    • libssh
  • ssh
  • your apps

Debian official packages are dbus libssh-2 socat
gabriel is not part of Debian yet (but I’ve build one for myself)

Knowledge (Actors curriculum)

In this section I will describe the basics about the tools we are going to use.

DBus. Extracted from DBus page:

D-Bus is a message bus system, a simple way for applications to talk to one another. In addition to interprocess communication, D-Bus helps coordinate process lifecycle; it makes it simple and reliable to code a “single instance” application or daemon, and to launch applications and daemons on demand when their services are needed.

D-Bus supplies both a system daemon (for events such as “new hardware device added” or “printer queue changed”) and a per-user-login-session daemon (for general IPC needs among user applications). Also, the message bus is built on top of a general one-to-one message passing framework, which can be used by any two apps to communicate directly (without going through the message bus daemon). Currently the communicating applications are on one computer, or through unencrypted TCP/IP suitable for use behind a firewall with shared NFS home directories.

Gabriel is a simple utility to enable D-Bus clients to connect to a D-Bus daemon running on a remote machine, through SSH.
This is the main piece of this puzzle. If you are interested in understanding how it works you should take a look at socat and libssh. As I’ve had to take a look at code, and make some modifications, you should read it as a punishment.

Extracted from socat man page:

socat – Multipurpose relay (SOcket CAT)
socat is a command line based utility that establishes two bidirectional byte streams and transfers data between them. Because the streams can be constructed from a large set of different types of data sinks and sources (see address types), and because lots of address options may be applied to the streams, socat can be used for many different purposes. It might be one of the tools that one ‘has already needed´.

Libssh. Extracted from libssh page:

The ssh library was designed to be used by programmers needing a working SSH implementation by the mean of a library. The complete control of the client is made by the programmer. With libssh, you can remotely execute programs, transfer files, use a secure and transparent tunnel for your remote programs. With its Secure FTP implementation, you can play with remote files easily, without third-party programs others than libcrypto (from openssl).

You should know about ssh and about your application.

Architecture

Local host will run gabriel and your application.
Remove host will need a running ssh server, a running dbus server and will need socat installed and ready to use.
We need to run gabriel, that will act as a server that will connect our host to the remote host through SSH. After that gabriel will use this SSH connection to intercommunicate our local application with remote DBus applications by using socat.

Remote DBus communication Architecture
Remote DBus communication Architecture

Howto (Main action)

At the moment I’ve only achieved to connect a process using session-bus, I’m still testing until I get connection through system-bus which was my initial purpose.
After reading next information, you will be able to connect using session bus and system bus.

As I commented somewhere else, I’ve made some modifications on gabriel code. I needed some common parameters as SSH port (my virtualbox testing environment ), better help explanations or add a verbose output.
Gabriel establish a connection with the remote ssh and by socat commands it communicates with the remote DBus “environment”. You should administrate ssh parameters and Dbus parameters to gabriel.

We have to put special attention to -d, –bus-address=BUS_ADDRESS because this info must be gotten from the REMOTE machine.
That address is the one used by processes to communicate through DBUS. It’s something “internal” and automatically done when you use DBus api/library. I’m going to show you where to get it.

DBUS_SESSION_BUS_ADDRESS, DBUS_SYSTEM_BUS_ADDRESS, DBUS_SYSTEM_BUS_DEFAULT_ADDRESS

Again, this info should be gotten from REMOTE machine.
At the moment I don’t know any nice command where to get this info.
We have two main options of DBus buses. System and Session (more info in DBus page).
If you need SESSION bus address, you can choose what it better fits you:

  • You can can get it from process environment
  • You can stole it from any other process suspicious from being involved in DBus activities…
  • You can create your own dbus-daemon (which, actually, I don’t know if it uses it’s own BUS_ADDRESS)

If you need SYSTEM bus address, you can choose what it better fits you:

  • You can can get it from process environment. If it’s not defined, take a look at /etc/dbus-1/system.conf where you should locate a string like <listen>unix:path=/var/run/dbus/system_bus_socket</listen>
  • You can stole it from any other process suspicious from being involved in DBus activities…

Examples:

This command gives you a dbus-daemon in your session with the one you can contact.

Howto (Main action): Back to local host

Those ugly unix:stri:ngs/asdkaj/numbers we have seen is what we need for -d, –bus-address=BUS_ADDRESS.
See a session example:

See a system example:

The moment we have or gabriel server running we (may have nothing) need to set DBUS_XXX_BUS_ADDRESS. Many apps would use, or have, this environment variable to connect to a DBus instance and intercommunicate with other process.
This is is easy, DBUS_XXX_BUS_ADDRESS should be the address gabriel shows few instants after being launched.
When we have defined this environment variable (in command line) we can execute our app, and it will happily communicate with the remote DBus world.
Example:

dbus-browser is a program that uses a session bus.

Curiosity: DBus protocol messages interchanged

Modifying a couple of lines in gabriel can let you see DBus raw protocol messages. It’s a didactic info.
If you enable verbose code at least at level 2, you will get raw DBus protocol messages.

My modifications and hacks

Code will be publish under GLKM project page.

Links and references
  • dbus site
  • gabriel site
  • socat site
  • libssh site
  • reference 1. (informational note, it had implied jumping into gabriel, libssh, and dbus code and testing with a virtualbox machine)
  • reference 2. (personal note, take a look at “Securing traffic between two socat instances using SSL” article in socat page)

Howto access a Virtualbox guest machine throught ssh (or how to port forwarding)

This is not an original article (see references), it’s just an archived one “por si las moscas”.

Introduction

By default, the network connection in Virtualbox is set to NAT, that is every packet coming from the Guest machine is modified so that it seems as it has come from the Host machine. In this way it’s easy for the Guest machine to connect to all the rest of the network (the Internet included) but nobody can start a connection with the Guest Machine since it’s hidden behind the Host one.

So, if you want to use a server service in your Guest machine (i.e. Apache or SSH) you have two choices:

  1. pass to Virtualbox Host network connection;
  2. make Virtualbox forward all the packets arriving to a certain port of the Host machine.

This article will describe how to do the latter, in particular in the case of the ssh server. This is an interesting case because it allows you to simulate very well a quite common condition: connecting to a remote Linux headless machine.

We have a guest machine with a running ssh server which accepts connections on the TCP port 22.
Our goal is to make any packet arriving at a given TCP port (i.e. 2222) of the host machine, to be forwarded to the TCP port 22 of the guest machine.
Fortunately, there is Virtualbox command which permits to do it almost instantly: VBoxManage.

Configuration needed

In our case we will use Debian as the guest machine name (quote in case guest machine name contains spaces), here are the commands that you have to type in the host machine (real one) console as your user (if you use another user it will try to look for its virtual machines):

user@machine $ VBoxManage list vms
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
Name: debian
Guest OS: Linux 2.6
UUID: 814e25f4-451e-4582-8d34-71a1cd437cdd
Config file: /mnt/extra/virtualizacion/virtualbox/debian/debian.xml
Memory size: 195MB
[…]
user@machine $ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort” 2222
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
user@machine $ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort” 22
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
user@machine$ VBoxManage setextradata debian “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol” TCP
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.

HostPort,GuestPort,Protocol are just string examples and can be changed without any consequence. Just remember your choices if you want to remove them later.
I guess you should be careful with pcnet substring, as maybe it should have something to see with your selected network card type.

Testing

Once you have typed the above commands, you need to close the guest machine (a reboot won’t be sufficient), restart it and then connect via ssh with:

anyuser@machine$ ssh -l >user< -p 2222 localhost

Replace localhost with the host machine ip address if you are connecting from another computer.

By the way, you can check which customizations have been already set for your Guest Machine with VBoxManage by typing:

user@machine$ VBoxManage getextradata debian enumerate
VirtualBox Command Line Management Interface Version 1.5.6_OSE
(C) 2005-2008 innotek GmbH
All rights reserved.
Key: GUI/LastWindowPostion, Value: 0,6,1028,820
Key: GUI/Fullscreen, Value: off
Key: GUI/AutoresizeGuest, Value: off
Key: GUI/LastCloseAction, Value: powerOff
Key: GUI/SaveMountedAtRuntime, Value: yes
Key: GUI/Seamless, Value: off
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort, Value: 2222
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort, Value: 22
Key: VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol, Value: TCP

or remove one, for example “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort”, by setting it without any value:

user@machine$ VBoxManage setextradata “VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort”

Links and references

[GLKM] Subversion project stats measures with mpy-svn-stats

A simple command line tool called mpy-svn-stats give us nice graphics using svn xml output.

You can see an example GLKM project statistics

It’s quite easy to use, see:

user@machine:/var/www/svn-stats$ mpy-svn-stats -o ./ svn+ssh://localhost/var/lib/svn/glkm/
Will generate PIL graphs.
getting data
running command: “svn -v –xml log svn+ssh://localhost/var/lib/svn/glkm/”
[…ssh stuff…]
done
parsing data
done
calculating stats
done
writing data
done
Have 13 stats objects, 13 of them are wanted.
user@machine:/var/www/svn-stats$ ll
total 48
-rw-r–r– 1 user user 10379 2008-02-06 17:06 changed_paths_multi_author_graph.png
-rw-r–r– 1 user user 10268 2008-02-06 17:06 commits_group_multi_author_graph.png
-rw-r–r– 1 user user 11762 2008-02-06 17:06 index.html
-rw-r–r– 1 user user 10265 2008-02-06 17:06 log_message_length_group_multi_author_graph.png
user@machine:/var/www/svn-stats$

References and links

[GLKM] Funcionalidades

En esta entrada sólo me limito a escribir las principales funcionalidades/objetivos que ya pensé de dotar a mi PFC. Son un poco datos que debí haber publicado hace ya algún tiempo (para variar).
A priori no tengo porque implementarlo todo para poder terminar, ya que como se puede observar, terminan siendo muchas cosas.

Este futuro monitor gráfico de procesos del kernel linux será útil para depurar procesos tanto de la propia máquina como de una máquina remota.
La información monitorizada corresponde a las estructuras internas del sistema operativo, más concretamente, del sistema de ficheros con respecto a los procesos.

  • Orientado a una interfaz gráfica.
  • Monitorización en red.
  • Monitorización de procesos. Específicamente atributos de ficheros Reiser. Por supuesto la arquitectura va encaminada a extensiones que permitan ampliar las estructuras a monitorizar
  • Visualizar/Monitorizar grupos de procesos simultáneamente.
  • Estadísticas del uso de ficheros (opcional)
  • Tipos de monitorización: Tiempo real o asíncrona. En tiempo real se toman datos cada intervalo X de tiempo de forma constante. En el caso de monitorización asíncrona se toman datos en los instantes que se indiquen.
  • Integración con DBUS/HAL. En la medida de lo posible quiero llevar la comunicación entre la interfaz gráfica y el módulo del kernel por DBUS. A ver como sale.

No cabe duda que no quiero perder de vista al dtrace para Linux, systemtap. Systemtap realiza labores similares al fin y al cabo.
Para quien no conozca estas herramientas en los siguiente párrafos hago un breve resumen.
Dtrace y systemtap son muy similares, sólo que una está ligada a Solaris y la otra a Linux. Como en mi caso, monitorizar el núcleo del sistema operativo te cierne (un poco por definición) a ser una herramienta monoplataforma. Ambas proveen unos “lenguajes” propios que permiten extraer/monitorizar variables del sistema operativo.
Con dicha información se pueden realizar estadísticas y depuración como objetivo primordial.

PKG_PROG_PKG_CONFIG: command not found

Minutes ago I was trying to compile hal++ a wrapper for libhal developed by beep-media-player team.
While executing configure I’ve got this error:

./configure: line XXXXX: PKG_PROG_PKG_CONFIG: command not found

It was not expected because I did have autotools and pkg-config already installed. And as incredible as I had compiled without any error hal++ in other Debian testing/unstable machine (I was in a Debian testing).
So I have investigated a bit with half success. I’ve found a peace of autotools code which have been useful:

# Check for pkg-config manually first, as if its not installed the
# PKG_PROG_PKG_CONFIG macro won’t be defined.
#
AC_CHECK_PROG(have_pkg_config, pkg-config, yes, no)
#
if test x”$have_pkg_config” == xno; then
AC_MSG_ERROR(pkg-config is required to install this program)
fi
#
PKG_PROG_PKG_CONFIG

It’s only a couple of verifications, which are not introducing anything new, but it was incredibly worked.

Now, I’ll try libhal++ to see if I can do what I want in C++.

intltoolize: cannot copy '/usr/share/intltool/Makefile.in.in' to 'po/Makefile.in.in'

Un pequeño problema para el que no he encontrado la solución por más que he buscado.
El problema ha surgido ahora que estoy usando las autotools para automatizar toda la compilación, linkado y distribución de GLKM.

El caso es que al ejecutar el autogen.sh que tengo preparado me salía un mensaje, entre la maraña que sueltan estas herramientas, del tipo:

user@einstein:~/svn/pfc/trunk/gui$ ./autogen.sh

cp: no se puede crear el fichero regular «po/Makefile.in.in»: No existe el fichero o el directorio

Acotando el comando que fallaba lo ejecuté en modo depuración, para saber que hacía exactamente.

user@einstein:~/svn/pfc/trunk/gui$ intltoolize –copy –force –automake –debug
intltoolize: enabling shell trace mode
+ test -f configure.ac
+ configure=configure.ac
+ files=’intltool-extract.in intltool-merge.in intltool-update.in po/Makefile.in.in’
+ auxdir=.
++ egrep ‘^AC_CONFIG_AUX_DIR’ configure.ac
+ auxdirline=’AC_CONFIG_AUX_DIR([build-aux])’
+ test -n ‘AC_CONFIG_AUX_DIR([build-aux])’
++ echo ‘AC_CONFIG_AUX_DIR([build-aux])’
++ sed ‘s/^AC_CONFIG_AUX_DIR(([^)]*)).*$/1/’
+ auxdir='[build-aux]’
+ test ‘[build-aux]’ = ‘AC_CONFIG_AUX_DIR([build-aux])’
++ echo ‘[build-aux]’
++ sed ‘s/^[(.*)]$/1/g’
+ auxdir=build-aux
+ case “$auxdir” in
+ test -z yes
++ pwd
+ cur=/home/user/svn/pfc/trunk/gui
+ test build-aux ‘!=’ .
+ test -z yes
+ cd build-aux
+ for file in ‘$files’
+ test -f intltool-extract.in
+ test -z yes
+ rm -f intltool-extract.in
+ test -n ”
++ basename intltool-extract.in
+ cp -f /usr/share/intltool/intltool-extract.in intltool-extract.in
+ :
+ for file in ‘$files’
+ test -f intltool-merge.in
+ test -z yes
+ rm -f intltool-merge.in
+ test -n ”
++ basename intltool-merge.in
+ cp -f /usr/share/intltool/intltool-merge.in intltool-merge.in
+ :
+ for file in ‘$files’
+ test -f intltool-update.in
+ test -z yes
+ rm -f intltool-update.in
+ test -n ”
++ basename intltool-update.in
+ cp -f /usr/share/intltool/intltool-update.in intltool-update.in
+ :
+ for file in ‘$files’
+ test -f po/Makefile.in.in
+ rm -f po/Makefile.in.in
+ test -n ”
++ basename po/Makefile.in.in
+ cp -f /usr/share/intltool/Makefile.in.in po/Makefile.in.in
cp: no se puede crear el fichero regular «po/Makefile.in.in»: No existe el fichero o el directorio
++ basename po/Makefile.in.in
+ echo ‘intltoolize: cannot copy ”’/usr/share/intltool/Makefile.in.in”’ to ”’po/Makefile.in.in””
intltoolize: cannot copy ‘/usr/share/intltool/Makefile.in.in’ to ‘po/Makefile.in.in’
+ status=1
+ exit 1
+ exit 1

Estoy usando un directorio para guardar los archivos de autoconf (config-aux) y tener la estructura de directorios y archivos ordenadita.
Ahí radicaba el problema. Así que con crear un directorio llamado po bajo ese directorio ha desaparecido el error.

user@einstein:~/svn/pfc/trunk/gui$ cd config-aux/
user@einstein:~/svn/pfc/trunk/gui/config-aux$ mkdir po
user@einstein:~/svn/pfc/trunk/gui/config-aux$ ll
total 412K
-rwxr-xr-x 1 user user 44K 2007-04-20 04:09 config.guess*
-rwxr-xr-x 1 user user 32K 2007-04-20 04:09 config.sub*
-rwxr-xr-x 1 user user 18K 2007-06-07 21:28 depcomp*
-rwxr-xr-x 1 user user 13K 2007-06-07 21:28 install-sh*
-rw-r–r– 1 user user 23K 2007-06-14 16:53 intltool-extract.in
-rw-r–r– 1 user user 36K 2007-06-14 16:53 intltool-merge.in
-rw-r–r– 1 user user 28K 2007-06-14 16:53 intltool-update.in
-rw-r–r– 1 user user 193K 2006-03-11 19:49 ltmain.sh
-rwxr-xr-x 1 user user 11K 2007-06-07 21:28 missing*
-rwxr-xr-x 1 user user 2,0K 2007-06-14 16:52 mkinstalldirs*
drwxr-xr-x 2 user user 48 2007-06-14 17:51 po/

First Tuesday Madrid

Mi sangre emprendedora empieza a fluir.

No sólo sigo bastante de cerca los comentarios de algunos blogs de marketing, publicidad y emprendedores. Ahora también “participo”.
De momento me siento confuso. Como si fuese a un sitio dónde (de momento) no me llaman pero en el que no tengo pánico a nada.

En el First Tuesday Madrid que es dentro de unas horas en Capitan Haya voy a empezar a conocer que se cuece en el ambiente del mundillo emprendedor.
De momento se que voy a ver a uno que conozco y que hace tiempo que no veo y que voy a conocer a otro al que leo. Y supongo que alguno más.

Dudo que participe mucho, pero observaré e intentaré aprender al máximo, que es lo que mejor se hacer. Para cuando me atreva a lanzarme a la piscina.
Aunque lo primero es lo primero. El PFC/TFC (GLKM) sigue su rumbo.

En unas horas más impresiones.

svn: can not read file '/var/lib/svn/glkm/db/revprops/5': End of file found

Si tras ejecutar subversion con un svn co, o un svn update el resultado de la operación os luce con un bonito:

user@machine:~/svn/pfc/trunk/gui/src$ svn update
svn: No se pudo leer el archivo ‘/var/lib/svn/glkm/db/revprops/5’: End of file found

El problema es que el archivo en cuestión, llamémosle forunculo, contiene algún error.
Hasta aquí todo esto lo puede haber pensado cualquier hijo de vecino.

En mi caso salía porque había tocado a mano ese archivo para modificar el log de subversion en ese commit/revision, número 5. Le había quitado algunas letras y lo había dejado tal cual.
Cuando intenté posteriormente obtener esa versión para hacer unas pruebas me encontré con el fallo.

Estos archivos de propiedades de cada versión (eso deduzco por el nombre) contienen unos campos de control de longitud, algo muy cutre, pero muy sencillo. Para arreglar el problema hay que contar de manera exacta los caracteres ocupados (yo resté, menos mal que eran pocos).

Un ejemplo similar al caso real (Ojo, antes del END deberían aparecer 2 líneas vacías, en el ejemplo no se muestran):

user@machine-svnserver:/var/lib/svn/glkm/db# cat revprops/5
K 10
K 10
svn:author
V 9
wodofredo
K 8
svn:date
V 27
2007-05-11T17:44:13.460401Z
K 7
svn:log
V 19
2007-05-11 Enrique
END

aristoteles:/var/lib/svn/glkm/db# diff revprops/5 revprops/5~
< V 24 < 2007-05-11 Enrique jdfgj --- > V 19
> 2007-05-11 Enrique