Debian Wheezy with KDE

The right way to install KDE on Debian Wheezy is to start the right installer by choosing “KDE desktop” under Alternate Desktop Environments under Advanced Options in the CD boot menu.
If you haven’t done that, you’ll end with Gnome. In this case here is the cure:

sudo apt-get autoremove --purge gnome*
sudo apt-get install kde-standard
pixelstats trackingpixel
Posted in Howtos, Uncategorized | Leave a comment

Apple OS X virtualization with VirtualBox

A long time ago virtualizing Apple OS X was forbidden by the End User License Agreement, with the exception of Server versions of the operating system. Consequently Oracle (the maintainer of VirtualBox) was not very supportive, and many replies on the end user forum for VirtualBox still point to the then default answer.

Countless hackers spent endless nights to work around this issue, and thousands of posts offer tricks and step-by-step guides that are now mostly obsolete. Also, many of these refer to the process of virtualizing OS X on non-Apple hardware, which is illegal (so called “hackintosh”).

Two years ago, with OS X Lion (10.7) Apple started allowing users to run up to two additional instances of OS X on the same machine without a need for extra licenses. It took some time to the VirtualBox developers to pick up this fact and enhance their support for OS X guests, but with VirtualBox 4.3 released on October 15th, 2013 things start to be useable.

More precisely the following is a report on running Apple Max OS X 10.7, 10.8 and 10.9 as guests under VirtualBox 4.3.2 on a 10.7 host.

To create 10.7 and 10.8 guests, you need to extract the InstallESD.dmg from the restore image or from the installer downloaded from the iTunes store. There are step-by-step guides around to do that. You do not need to patch the InstallESD.dmg in any way.

Just go ahead and create a new VM with the correct OS type (i.e. Mac OS X) and version (i.e. Mac OS X 10.7 Lion (64 bit) or Mac OS X 10.8 Mountain Lion (64 bit)). Pick up all the default options or change RAM and disk sizes, then attach the InstallESD.dmg file to the VM’s CD/DVD.

Before you launch it, if you like you can prepare for setting the VM screen resolution to 1440×900 by executing these commands in a terminal of the host:

VBoxManage setextradata "OSX_108" "CustomVideoMode1" "1440x900x32
VBoxManage setextradata "OSX_108" "GUI/CustomVideoMode1" "1440x900x32"
VBoxManage setextradata "OSX_108" VBoxInternal2/EfiGopMode 4
VBoxManage setextradata "OSX_108" VBoxInternal2/UgaHorizontalResolution 1440
VBoxManage setextradata "OSX_108" VBoxInternal2/UgaVerticalResolution 900

(these may well be quite redundant but they’re guaranteed to work).

The install process should be slower than on bare metal, but smooth with the exception of some occasional hangs on booting (just kill the VM and retry).

When you are done, remember to turn off energy saving (set timeout to “Never”) in the system preferences as recommended here.

To complete the procedure for setting the VM screen resolution to 1440×900, execute the command:

sudo vi /Library/Preferences/SystemConfiguration/com.apple.Boot.plist

in the guest operating system, and inside the

<dict>
...
</dict>

section insert:

<key>Graphics Mode</key>
<string>1920x1080x32</string>
<key>Kernel Flags</key>
<string>"Graphics Mode"="1920x1080x32"</string>

The matter with 10.9 is different because VirtualBox does not seem to be able to start the VM from the unpatched InstallESD.dmg, even if the correct OS version is chosen (Mac OS X 10.9 Mavericks (64 bit)). The method I used was to clone a 10.8 VM and launch the installer from there.

Mandatory screenshot:

bbb

pixelstats trackingpixel
Posted in Howtos, Uncategorized | Leave a comment

Doing RPC in 2013

Some history

18 years ago I bought a book titled “Power programming with RPC”; I was surprised to see that the 1992 book is still available now ! After reading it, at the time I was really excited about RPC (Remote Procedure Call) which is indeed a useful thing to have.

But I was also very confused about how to actually implement it. Many solutions have been proposed over the years, from Sun’s Open Network Computing (ONC) which is the technology presented in that book: CORBA (who still remembers about that ?), Microsoft DCOM and .NET, SOAP, XML-RPC, JSON-RPC, Java RMI (Remote Method Invocation), the Internet Communications Engine (ICE), Autobahn

None of these seem to fit my requirements:

  • portability: it should work with almost any combination of client-server choosen among HTML5 browser, browser OS, mobile OS (just to name three of them: Android, iOS, WP8), desktop OS (just to name three of them: Windows, Max OS X, Linux)
  • patent-free
  • open source, non-pure-GPL, C++ implementations available
  • defaulting to a low-overhead, zero-configuration protocol when the RPC is invoked between a client and a server which run locally as separate processes on the same device.

In exchange for those, I can live without discovery and an Interface Description Language (IDL).

Enter Qt and QJsonRpc

But wait, since RPC is basically calling remote procedures on a server by sending over strings, Qt’s meta object system is a good basis to build on: it is portable, it has reflection to pick up the method to call from the string, it provides signals-slots to implement call-backs, it has a variant type. Qt also provides interexchangable QTcpSocket for two-way connections over the network and QLocalSocket for local connections (QLocalSocket uses a named pipe on Windows and a local domain socket on Unix).

While musing about those facts, I stumbled upon QJsonRpc, a Qt implementation of the JSON-RPC protocol available here. It is compatible with both Qt4 and Qt5, and it is licensed under the LGPLv2.1. It is developed by devonit, an innovative firm which BTW has just introduced a smart new device which plugs into the HDMI port of any monitor and transforms it into a thin client.

There is not much documentation around on QJsonRpc so to understand what this thing does, let’s try it out – here is how I did on Debian wheezy with Qt4 and on jessie with Qt5;  to set-up on a fresh install of Debian jessie, do this:

sudo apt-get install qtbase5-dev g++

Then:

wget https://bitbucket.org/devonit/qjsonrpc/get/81c851a5c77a.zip
unzip 81c851a5c77a.zip
cd devonit-qjsonrpc-81c851a5c77a/

If you are using Qt4, issue:

qmake

whereas on jessie to use Qt5, do:

/usr/lib/i386-linux-gnu/qt5/bin/qmake)

Then:

make
cd tests/manual
cat > ./localserver/runserver
#!/bin/bash
/lib/ld-linux.so.2 --library-path ../../src ./localserver/server
^D
chmod u+x ./localserver/runserver
cat > ./qjsonrpc/runqjsonrpc
#!/bin/bash
/lib/ld-linux.so.2 --library-path ../../src ./qjsonrpc/qjsonrpc $@
^D
chmod u+x ./qjsonrpc/runqjsonrpc
cat > ./localclient/runclient
#!/bin/bash
/lib/ld-linux.so.2 --library-path ../../src ./localclient/localclient
^D
chmod u+x ./localclient/runclient

Now start the local server:

./localserver/runserver

then open a new console and start the local client:

./localclient/runclient

The local client will print this:

response received: QJsonRpcMessage(type=QJsonRpcMessage::Response, id=1, result=QVariant(, ) )
response received: QJsonRpcMessage(type=QJsonRpcMessage::Response, id=2, result=QVariant(, ) )
response received: QJsonRpcMessage(type=QJsonRpcMessage::Response, id=3, result=QVariant(, ) )
response received: QJsonRpcMessage(type=QJsonRpcMessage::Response, id=4, result=QVariant(QString, "Hello matt") )

whereas the localserver will print this:

void TestService::testMethod() called
void TestService::testMethodWithParams(const QString&, bool, double) called with parameters:
first: "one"
second: false
third: 10
void TestService::testMethodWithVariantParams(const QString&, bool, double, const QVariant&) called with variant parameters:
first: "one"
second: false
third: 10
fourth: QVariant(double, 2.5)
QString TestService::testMethodWithParamsAndReturnValue(const QString&) called

To better understand what is going on, look into the code ! For example the key client-side code for the last request is (let’s skip error handling for ease of understanding):

QJsonRpcSocket *m_client = new QJsonRpcSocket(socket, this);
QJsonRpcServiceReply *reply = m_client->invokeRemoteMethod("agent.testMethod");
reply = m_client->invokeRemoteMethod("agent.testMethodWithParamsAndReturnValue", "matt");
connect(reply, SIGNAL(finished()), this, SLOT(processResponse()));

and the call-back:

void LocalClient::processResponse() {
  QJsonRpcServiceReply *reply = static_cast<QJsonRpcServiceReply *>(sender());
  qDebug() << "response received: " << reply->response();
}

whereas the key server-side code is:

TestService service;
QJsonRpcLocalServer rpcServer;
rpcServer.addService(&service);
rpcServer.listen(serviceName);

and the server-side implementation of the testMethodWithParamsAndReturnValue method:

QString TestService::testMethodWithParamsAndReturnValue(const QString &name) {
  return QString("Hello %1").arg(name);
}

Now shut down the client by hitting ^C in the client console, then launch the command-line client:

./qjsonrpc/runqjsonrpc testservice agent.testMethod

this will print:

QVariant(, )

or on Qt5:

QVariant(Invalid)

whereas

./qjsonrpc/runqjsonrpc testservice agent.testMethodWithParamsAndReturnValue matt

will print:

QVariant(QString, "Hello matt")

The command-line interface is only able to connect to a local server, but as to the server / client, there is also a demo over TCP on port 5555.
If we diff the localclient (using Qt’s QLocalSocket) and the tcpclient (using QTcpSocket), there are very few differences, the main one is:

< QLocalSocket *socket = new QLocalSocket(this);
< socket->connectToServer(serviceName);
---
> QTcpSocket *socket = new QTcpSocket(this);
> socket->connectToHost(QHostAddress::LocalHost, 5555);

where serviceName is a QString obtained by calling QDir::absoluteFilePath on the file /tmp/testservice.

Diffing the server also shows very few differences, the main one is:

< if (!rpcServer.listen(serviceName)) {
---
> if (!rpcServer.listen(QHostAddress::LocalHost, 5555)) {

and the testservice implementation is actually the same !

Enter Websockets and QWebSockets

Now connecting over a WAN over port 5555 with a custom protocol calls for firewall troubles. Your IT people will complain they have to open new ports and your customer’s antivirus will harass them. Also no other application on earth will talk to your server.

Better use Websockets, a web technology providing full-duplex communications channels over a single TCP connection over port 80, configured as an upgrade to the HTTP protocol, and compatible with a number of browsers (Google Chrome, Internet Explorer, Firefox, Safari and Opera) which can act as clients if you develop an HTML5 interface.

And lo ! I stumbled upon QWebSockets, a pure Qt implementation of client and server WebSockets; it has no other dependencies that Qt5 and it is licensed under LGPL v2.1. One painful limitation is that it currently lacks the WSS protocol (the secure version of the ws:// protocol); but since it’s open-source, maybe somebody will care to implement it !

Here is how I tried it out on Debian jessie with Qt5:

wget https://codeload.github.com/KurtPattyn/QWebSockets/legacy.tar.gz/master
tar xzf KurtPattyn-QWebSockets-v0.9.0-97-g2f6831e.tar.gz
cd KurtPattyn-QWebSockets-2f6831e
/usr/lib/i386-linux-gnu/qt5/bin/qmake
make

now let’s test the provided example:

cd examples
./echoserver/echoserver

prints:

Echoserver listening on port 1234

then in a new console:

./echoclient/echoclient

prints:

Websocket connected
Message received: "Hello, world!"

Both libraries are LGPL licensed, so it would be great to set up a QJsonRpcWebServer based on the QJsonRpcTcpServer from QJsonRpc, to use the QWebSocketServer from QWebSockets in place of Qt’s builtin QTcpServer.

pixelstats trackingpixel
Posted in Philosophy, Uncategorized | 1 Comment

Reduce doxygen footprint on debian

When you install doxygen on a recent debian OS, it will also install doxygen-latex, which bloats the footprint (on 32-bit wheezy) from 7.5 to 963 MB.
To avoid that once and for all, pin the doxygen-latex package with a negative priority, thereby preventing it from ever installing:

sudo cat > /etc/apt/preferences.d/doxygen-latex
Package: doxygen-latex
Pin: version *
Pin-Priority: -1
^d

To be precise, this trick lowers the requirement for doxygen from 633 MB download and 963 MB disk space down to 2.5 MB download and 7.5 MB disk space.

pixelstats trackingpixel
Posted in Howtos | 2 Comments

Disable an account in Thunderbird / Icedove

If you have an email account which has been deactivated, you want Thunderbird / Icedove to stop checking for new messages each time. But you do not want to completely remove the account because you like to keep the old messages and maybe it’ll be reactivated soon.

So do this (tested in Icedove 17.x): Go to Menu – Edit – Preferences, Advanced section, General Tab, click on Config Editor button.

Enter mail.server.server*.name as search string and find the number corresponding to the email account you want to disable; let’s assume it’s the number 5.

Now enter mail.server.server5.check_new_mail (replace 5 with the actual number that works for you) and set that property to false.

Done !

pixelstats trackingpixel
Posted in Howtos | Leave a comment

The quest for a decent Android RSS aggregator

Do you pick posts to read on your 7″ Android tablet from a few RSS feeds ? Do you need an Android app to aggregate them, download the posts for offline reading and present them in a clean, lightweight list ?

With 700000 apps available on the Google app store for Android, you think that would be a solved problem. You also expect that firing your favorite search engine with a query like “RSS feed aggregator offline android” would give you a few independent reviews to help you find the right app.

Actually not. The query will return countless biased and ad-ridden reviews, pointing you to a collection of equally ad-ridden bloatware, designed to involve you in some random social network thingy.

I have NO FRIENDS ! I want NO ADS ! I have NO MONEY !

Compare the results of the query “RSS feed aggregator offline debian KDE“: it takes you a few clicks to find Akregator, the best RSS aggregator you can get for desktop use. Delights of a corporate-free ecosystem where people create great open source software without economic incentives !

Anyway going back to Android, stay clear from all the suggested apps and go for sparse rss: it’s open source and does what it should without pain.

Mandatory screenshot:

Screenshot_2013-04-09-11-16-00

pixelstats trackingpixel
Posted in Howtos, Rants | Leave a comment

Subscribing to trac’s RSS feed

With trac version 0.11.7 we hit a bug where requesting the timeline RSS triggers a UnicodeDecodeError. This can be fixed by changing line 224 of the file /usr/share/pyshared/buildbot/status/web/feeds.py as follows:

unilist.append(unicode(line,'utf-8','replace'))

where we exploit the more recent resilient version Python’s unicode function: with the ‘replace’ option the official Unicode replacement character, U+FFFD, is used to replace input characters which cannot be decoded and no exception is thrown.

But even if all works well with generating the trac‘s RSS feed, subscribing it can be tricky if trac authentication is set up in the standard way: this involves a redirect to fictitious trac/login page, which installs a certain cookie. RSS readers and aggregators are not able to get this cookie and basically hit a 403 (forbidden) error.

The advised procedure described here i.e. using the HttpAuthPlugin, may work for you, but it did not work here.

What does work here is downloading the trac RSS feed with wget. To achieve that, you need to visit the trace/login page first, with the correct username and password, just to store the cookie (you can discard the downloaded login page); next visit the RSS link proper without supplying username and password:

wget --keep-session-cookies --save-cookies a -O /dev/null --user=username --password=password --no-check-certificate 'https://server/trac/login'
wget --load-cookies a -O trac.rss --no-check-certificate 'https://server/trac/timeline?ticket=on&milestone=on&max=50&author=&daysback=90&format=rss'

adapt the code above to your installation by replacing the server, username and password appropriately.

Enjoy !

pixelstats trackingpixel
Posted in Howtos, Rants | Tagged | Leave a comment

Howto output XLSX files from C++

The title of this post should really be: Howto output Office Open XML (OpenXML) spreadsheet files from native C++ with as little hassle and unwanted dependencies as possible.

We will not discuss the many other options available, both commercial and not. What we want is a low-level (no .NET) C++ (no C#) API to write (no read) XLSX files, with support for:

  • multiple sheets
  • sparse sheets (one cell here, one cell there)
  • UTF-8 encoded strings into cells
  • split panes (horizontal, vertical and both horizontal and vertical)
  • basic formatting (plain, bold, “0.00″ and bold dates “yyyy/mm/dd”)
  • setting column widths.

The starting point was the excellent libOPC open source library by Florian Reuter, a cross-platform, open source, standard C99-based implementation of Part II (OPC) and Part III (MCE) of the ISO/IEC 29500 specification (OOXML).
As mentioned here, the library does not expose a specialized API to create Excel files yet, however there is an opc_generate tool which will create all the boilerplate code for you. Starting with one such boilerplate source file, we produced a 600-SLOC C++ API that only requires libOPC and boost::variant (include only) and allows you to write code such as:

#include "Xlsx.h"

int main() {
  Xlsx xlsx("test.xlsx");
  xlsx.setAppVersion("00.0001");
  xlsx.setAppName("Test producer");

  xlsx.addWorksheet("one");
  xlsx.addWorksheet("two");

  xlsx.sheet(0).set(20, 10, std::string("(20,10)"), 3); // plain format

  xlsx.sheet(1).set(0, 0, 1.2345, 4); //  "0.00"
  xlsx.sheet(1).set(0, 1, std::string("(0,1)"), 1); // bold
  xlsx.sheet(1).set(0, 2, std::string("(0,2)"), 1); // bold
  xlsx.sheet(3).set(0, 4, 45000, 2); // bold "yyyy/mm/dd"
  xlsx.sheet(1).set(2, 3, std::string("(2,3)"), 1); // bold
  xlsx.sheet(1).set(2, 4, std::string("012345678901234567890"), 3); // plain

  xlsx.sheet(1).setColumnWidth(5); // for all columns
  xlsx.sheet(1).setColumnWidth(0, 10); // override for column 0

  xlsx.sheet(1).setSplit(1, 1);

  return 0;
} // main

This archive contains the source code for the XLSX API and prebuilt libOPC (based on libopc-20526) for Debian Wheezy i386. To build and run on this platform (on other platforms you will have to configure and build libOPC first):

cd xlsx/src
bjam
../bin/src/gcc-4.7/debug/XlsxTest

The produced files can be opened fine with LibreOffice 3.5 and Microsoft Excel 2013 preview. They pass validation with this validator with one minor error (it claims http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties should have been http://schemas.openxmlformats.org/officedocument/2006/relationships/metadata/core-properties); the Open XML SDK 2.5 Productivity Tool provided by Microsoft reports “no validation error”.

xlsx

A sample produced file can be downloaded here.

One weak point of the XLSX API (apart form missing all the exciting stuff like pivot tables, shared strings, charts, named ranges, shapes, protection, encryption, data validation, conditional formatting … you name it) is the format support. Sorry, these 4 are the formats WE needed ! If you need more formats, create a sample XLSX file with all your required formats, generate the boilerplate code with the opc_generate tool then copy the content of the create_xl_styles_xml function. You will then have access (via those funny integer handles) to all your styles.

pixelstats trackingpixel
Posted in C++, Howtos | Leave a comment

Howto diff XML files or assemblies thereof such as OpenXML or ODF files

Office Open XML or Open Document Format for Office Applications (ODF) are both zipped, XML-based file formats, that can be extracted to directories.

Diffing XML files is tricky because the common diff tools may be confused by different line endings (LF vs CR-LF) or missing line ending, and whitespace and indentation.

We have this recipy here to compare two directories that contain XML files:

unzip A.xlsx -d A
unzip B.xlsx -d B
find A/ -type f | xargs fromdos
find B/ -type f | xargs fromdos
find A/ -type f | xargs xmlindent -w
find B/ -type f | xargs xmlindent -w
find A/ | grep '~' | xargs rm
find B/ | grep '~' | xargs rm
kdiff3 A/ B/
pixelstats trackingpixel
Posted in Howtos | Leave a comment

git push errors “insufficient permission for adding an object to repository” and “unpack failed: unpack-objects abnormal exit”

We occasionally have this error when pushing to a git repository:

$ git push
Counting objects: 7, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.14 KiB, done.
Total 4 (delta 3), reused 0 (delta 0)
error: insufficient permission for adding an object to repository database ./objects

fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To zzzzzz@xxxxx.libpf.com:/var/git/cases.git
 ! [remote rejected] test_qqqqq -> test_qqqqq (n/a (unpacker error))
error: failed to push some refs to
'zzzzzz@xxxxx.libpf.com:/var/git/cases.git'

This hints to some server-side permission problem. This is how we troubleshooted it.

First we scan the repository premissions:

find /var/git -type f | xargs ls -l  | awk '{print $1}' | sort | uniq

We had set the setgid for all directories in /var/git (“Setting the setgid permission on a directory (chmod g+s) causes new files and subdirectories created within it to inherit its group ID, rather than the primary group ID of the user who created the file (the owner ID is never affected, only the group ID). Newly created subdirectories inherit the setgid bit.“)  so we had expected all files to be “-rwxrwx—”, but this what we got:

-r--r--r--
-rw-r--r--
-rwxrwx---

Somebody had been writing files to the repositories with the wrong permissions ! But who ? This is how we found out:

find /var/git -type f | xargs ls -l  | grep 'r--'

A long list of files came out with the wrong permissions. The other users are not allowed to overwrite these files, potentially causing similar errors in the future.

The problem can be brute-force fixed with this command:

cd /var/git
chmod -R 770 *

But this will not prevent it from happening again. The true solution is to set the git config variable “core.sharedRepository” for all repositories:

cd /var/git/aaaaaa.git
git config core.sharedrepository true

etc. ect.

Hope it helps !

pixelstats trackingpixel
Posted in Howtos | 5 Comments