The quest for a decent Android RSS aggregator

Do you pick posts to read on your 7″ Android tablet from a few RSS feeds ? Do you need an Android app to aggregate them, download the posts for offline reading and present them in a clean, lightweight list ?

With 700000 apps available on the Google app store for Android, you think that would be a solved problem. You also expect that firing your favorite search engine with a query like “RSS feed aggregator offline android” would give you a few independent reviews to help you find the right app.

Actually not. The query will return countless biased and ad-ridden reviews, pointing you to a collection of equally ad-ridden bloatware, designed to involve you in some random social network thingy.

I have NO FRIENDS ! I want NO ADS ! I have NO MONEY !

Compare the results of the query “RSS feed aggregator offline debian KDE“: it takes you a few clicks to find Akregator, the best RSS aggregator you can get for desktop use. Delights of a corporate-free ecosystem where people create great open source software without economic incentives !

Anyway going back to Android, stay clear from all the suggested apps and go for sparse rss: it’s open source and does what it should without pain.

Mandatory screenshot:

Screenshot_2013-04-09-11-16-00

pixelstats trackingpixel
Posted in Howtos, Rants | Leave a comment

Subscribing to trac’s RSS feed

With trac version 0.11.7 we hit a bug where requesting the timeline RSS triggers a UnicodeDecodeError. This can be fixed by changing line 224 of the file /usr/share/pyshared/buildbot/status/web/feeds.py as follows:

unilist.append(unicode(line,'utf-8','replace'))

where we exploit the more recent resilient version Python’s unicode function: with the ‘replace’ option the official Unicode replacement character, U+FFFD, is used to replace input characters which cannot be decoded and no exception is thrown.

But even if all works well with generating the trac‘s RSS feed, subscribing it can be tricky if trac authentication is set up in the standard way: this involves a redirect to fictitious trac/login page, which installs a certain cookie. RSS readers and aggregators are not able to get this cookie and basically hit a 403 (forbidden) error.

The advised procedure described here i.e. using the HttpAuthPlugin, may work for you, but it did not work here.

What does work here is downloading the trac RSS feed with wget. To achieve that, you need to visit the trace/login page first, with the correct username and password, just to store the cookie (you can discard the downloaded login page); next visit the RSS link proper without supplying username and password:

wget --keep-session-cookies --save-cookies a -O /dev/null --user=username --password=password --no-check-certificate 'https://server/trac/login'
wget --load-cookies a -O trac.rss --no-check-certificate 'https://server/trac/timeline?ticket=on&milestone=on&max=50&author=&daysback=90&format=rss'

adapt the code above to your installation by replacing the server, username and password appropriately.

Enjoy !

pixelstats trackingpixel
Posted in Howtos, Rants | Tagged | Leave a comment

Howto output XLSX files from C++

The title of this post should really be: Howto output Office Open XML (OpenXML) spreadsheet files from native C++ with as little hassle and unwanted dependencies as possible.

We will not discuss the many other options available, both commercial and not. What we want is a low-level (no .NET) C++ (no C#) API to write (no read) XLSX files, with support for:

  • multiple sheets
  • sparse sheets (one cell here, one cell there)
  • UTF-8 encoded strings into cells
  • split panes (horizontal, vertical and both horizontal and vertical)
  • basic formatting (plain, bold, “0.00″ and bold dates “yyyy/mm/dd”)
  • setting column widths.

The starting point was the excellent libOPC open source library by Florian Reuter, a cross-platform, open source, standard C99-based implementation of Part II (OPC) and Part III (MCE) of the ISO/IEC 29500 specification (OOXML).
As mentioned here, the library does not expose a specialized API to create Excel files yet, however there is an opc_generate tool which will create all the boilerplate code for you. Starting with one such boilerplate source file, we produced a 600-SLOC C++ API that only requires libOPC and boost::variant (include only) and allows you to write code such as:

#include "Xlsx.h"

int main() {
  Xlsx xlsx("test.xlsx");
  xlsx.setAppVersion("00.0001");
  xlsx.setAppName("Test producer");

  xlsx.addWorksheet("one");
  xlsx.addWorksheet("two");

  xlsx.sheet(0).set(20, 10, std::string("(20,10)"), 3); // plain format

  xlsx.sheet(1).set(0, 0, 1.2345, 4); //  "0.00"
  xlsx.sheet(1).set(0, 1, std::string("(0,1)"), 1); // bold
  xlsx.sheet(1).set(0, 2, std::string("(0,2)"), 1); // bold
  xlsx.sheet(3).set(0, 4, 45000, 2); // bold "yyyy/mm/dd"
  xlsx.sheet(1).set(2, 3, std::string("(2,3)"), 1); // bold
  xlsx.sheet(1).set(2, 4, std::string("012345678901234567890"), 3); // plain

  xlsx.sheet(1).setColumnWidth(5); // for all columns
  xlsx.sheet(1).setColumnWidth(0, 10); // override for column 0

  xlsx.sheet(1).setSplit(1, 1);

  return 0;
} // main

This archive contains the source code for the XLSX API and prebuilt libOPC (based on libopc-20526) for Debian Wheezy i386. To build and run on this platform (on other platforms you will have to configure and build libOPC first):

cd xlsx/src
bjam
../bin/src/gcc-4.7/debug/XlsxTest

The produced files can be opened fine with LibreOffice 3.5 and Microsoft Excel 2013 preview. They pass validation with this validator with one minor error (it claims http://schemas.openxmlformats.org/package/2006/relationships/metadata/core-properties should have been http://schemas.openxmlformats.org/officedocument/2006/relationships/metadata/core-properties); the Open XML SDK 2.5 Productivity Tool provided by Microsoft reports “no validation error”.

xlsx

A sample produced file can be downloaded here.

One weak point of the XLSX API (apart form missing all the exciting stuff like pivot tables, shared strings, charts, named ranges, shapes, protection, encryption, data validation, conditional formatting … you name it) is the format support. Sorry, these 4 are the formats WE needed ! If you need more formats, create a sample XLSX file with all your required formats, generate the boilerplate code with the opc_generate tool then copy the content of the create_xl_styles_xml function. You will then have access (via those funny integer handles) to all your styles.

pixelstats trackingpixel
Posted in C++, Howtos | Leave a comment

Howto diff XML files or assemblies thereof such as OpenXML or ODF files

Office Open XML or Open Document Format for Office Applications (ODF) are both zipped, XML-based file formats, that can be extracted to directories.

Diffing XML files is tricky because the common diff tools may be confused by different line endings (LF vs CR-LF) or missing line ending, and whitespace and indentation.

We have this recipy here to compare two directories that contain XML files:

unzip A.xlsx -d A
unzip B.xlsx -d B
find A/ -type f | xargs fromdos
find B/ -type f | xargs fromdos
find A/ -type f | xargs xmlindent -w
find B/ -type f | xargs xmlindent -w
find A/ | grep '~' | xargs rm
find B/ | grep '~' | xargs rm
kdiff3 A/ B/
pixelstats trackingpixel
Posted in Howtos | Leave a comment

git push errors “insufficient permission for adding an object to repository” and “unpack failed: unpack-objects abnormal exit”

We occasionally have this error when pushing to a git repository:

$ git push
Counting objects: 7, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 1.14 KiB, done.
Total 4 (delta 3), reused 0 (delta 0)
error: insufficient permission for adding an object to repository database ./objects

fatal: failed to write object
error: unpack failed: unpack-objects abnormal exit
To zzzzzz@xxxxx.libpf.com:/var/git/cases.git
 ! [remote rejected] test_qqqqq -> test_qqqqq (n/a (unpacker error))
error: failed to push some refs to
'zzzzzz@xxxxx.libpf.com:/var/git/cases.git'

This hints to some server-side permission problem. This is how we troubleshooted it.

First we scan the repository premissions:

find /var/git -type f | xargs ls -l  | awk '{print $1}' | sort | uniq

We had set the setgid for all directories in /var/git (“Setting the setgid permission on a directory (chmod g+s) causes new files and subdirectories created within it to inherit its group ID, rather than the primary group ID of the user who created the file (the owner ID is never affected, only the group ID). Newly created subdirectories inherit the setgid bit.“)  so we had expected all files to be “-rwxrwx—”, but this what we got:

-r--r--r--
-rw-r--r--
-rwxrwx---

Somebody had been writing files to the repositories with the wrong permissions ! But who ? This is how we found out:

find /var/git -type f | xargs ls -l  | grep 'r--'

A long list of files came out with the wrong permissions. The other users are not allowed to overwrite these files, potentially causing similar errors in the future.

The problem can be brute-force fixed with this command:

cd /var/git
chmod -R 770 *

But this will not prevent it from happening again. The true solution is to set the git config variable “core.sharedRepository” for all repositories:

cd /var/git/aaaaaa.git
git config core.sharedrepository true

etc. ect.

Hope it helps !

pixelstats trackingpixel
Posted in Howtos | Leave a comment

Howto link in boost::test on select platforms and get smaller test executables

The most useful boost libraries are include-only so it is handy to install just the boost headers on the select platforms (i.e. Windows) where it is cumbersome to install the compiled boost libs.

If you are using boost::test Unit Test Framework, you then need to include it directly into your test module. The penalty here is that this makes your executables bulky. On the other hand linking boost::test is easy on platforms where the compiled boost libs are easy to install, such as Debian.

We use this technique to link in boost::test on select platforms and get smaller test executables.

  1. We add this code to each test module, to control link/not linking boost::test based on an externally controllable macro “LINKTEST”:
    #define BOOST_TEST_MODULE test_name
    
    #ifdef LINKTEST
      #define BOOST_TEST_DYN_LINK
      #include <boost/test/unit_test.hpp>
    #else
      #include <boost/test/included/unit_test.hpp>
    #endif
    
    BOOST_AUTO_TEST_CASE(test_init) {
      ...
  2. We then add this code to the Jamfile, to link in boost::test only on linux platforms:
    project cases
      : requirements
        <target-os>linux:<define>LINKTEST
        <include>include ;
    
    ...

The executables when linked with boost::test will shrink by about 4,3 MB each in debug configuration, and by 0,8 MB in release.
This was tested on Debian 7.0 Wheezy 32 bit.

pixelstats trackingpixel
Posted in Howtos | Tagged | Leave a comment

Connecting LIBPF OPC to a Honeywell TPN Server via DCOM

The current DCS offering from Honeywell is Experion PKS, but there are still many plants around which use the previous system, Total Plant Solution (TPS).

On the TotalPlant Process Network (TPN) , additional applications are run on the Application Processing Platform (APP) node; one such applications can be the TPN Server, the real-time data server which provides OPC DA 2.0.x interface to read and write process tags.

If you want to run a LIBPF OPC instance that should connect to the Honeywell TPN Server, there is no need to run it on the APP node: you can use DCOM to connect from any other workstation on the same LAN as the APP node:

To configure the workstation that will run the LIBPF OPC client, follow these steps (tested with Windows XP SP3 client, Windows Server 2003 server):

  1. Save these lines in a file called “TPNserver.reg“:
    Windows Registry Editor Version 5.00
    
    [HKEY_CLASSES_ROOT\AppID\{ADF6AEBB-B0F1-11d0-8A01-00C04FC97D9D}]
    @="HCI_TPNServer - exe Server"
    "RemoteServerName"="enter_your_server_name_here"
    
    [HKEY_CLASSES_ROOT\CLSID\{ADF6AEBB-B0F1-11d0-8A01-00C04FC97D9D}]
    @="HCI_TPNServer - exe Server"
    "AppID"="{ADF6AEBB-B0F1-11d0-8A01-00C04FC97D9D}"
    
    [HKEY_CLASSES_ROOT\CLSID\{ADF6AEBB-B0F1-11d0-8A01-00C04FC97D9D}\ProgID]
    @="Hci.TPNServer"
    
    [HKEY_CLASSES_ROOT\Hci.TPNServer]
    @="HCI_TPNServer - exe Server"
    
    [HKEY_CLASSES_ROOT\Hci.TPNServer\CLSID]
    @="{ADF6AEBB-B0F1-11d0-8A01-00C04FC97D9D}"
    
    [HKEY_CLASSES_ROOT\Hci.TPNServer\OPC]
    @=""
  2. Load the file into the registry by double-clicking the TPNserver.reg file you just created.
  3. Perform the DCOM configuration steps explained here

Enjoy !

pixelstats trackingpixel
Posted in Howtos | Leave a comment

Detecting non-ASCII characters in a text file

Our internal coding standard for C++ source files dictates that 7-bit US-ASCII should be used for file encoding.

This decision is based on the fact that the current C++ standard (2003) limits characters that can be used in variable and type identifiers to ASCII letters. Although some compilers and the new (2011) C++ standard allow most Unicode code points in identifiers (basically whatever can be called a “letter” in the various scripts), the “same-glyph, different Unicode code-point syndrome” described here advises against that.

One could still allow non-ASCII characters in string constants and in comments, and this is tolerated by most modern compilers. But the decision was to be quite conservative in the current standard; in the future, as C++ 2011 is fully implemented, we might revise it.

The trouble is that sometimes non-ASCII characters sneak in, for example the euro sign , the degree symbol ° and the dash which looks so similar to the minus sign -.

Long story short, we needed an utility to detect non-ASCII characters in a collection of text (source) files. This utility is called checkAscii, and the C++ source code is:

/*
   @file checkAscii.cc
   @brief Detect non-ASCII characters in a text file
   @author (C) Copyright 2012 Paolo Greppi libpf.com
   @date 20120525
   @version 0.1

   no warranties whatsoever
   distribute freely and free of charge citing this:

http://wp.libpf.com/?p=626

*/

#include <iostream>
#include <fstream>
#include <cstdio>

int main(int argc, char *argv[]) {
  std::istream *in = NULL;
  std::ifstream inf;
  if (argc == 1) {
    in = &std::cin;
    std::cout << "Now checking stdin" << std::endl;
  } else if (argc == 2) {
    inf.open(argv[1]);
    if(!inf) {
      std::cerr << "Error opening input file !" << std::endl;
      return -1;
    }
    in = &inf;
    std::cout << "Now checking file " << argv[1] << std::endl;
  } else {
    std::cerr << "Only 0 or 1 argument !" << std::endl;
    return -1;
  }

  char c, bit8 = (1 << 7);
  int line(0), column(0), count(0);
  while ((c = in->get()) && (c != EOF)) {
    if (c == '\n') {
      ++line;
      column = 0;
    }
    if ((c & bit8) == bit8) {
      std::cout << "line: " << line + 1 << " column: " << column << " nonascii " << c << std::endl;       count++;     }     ++column;   }      if (argc > 1) {
    inf.close();
  }
  return count;
}

Usage is as follows:

cat mySourceFile.cc | checkAscii

or:

checkAscii mySourceFile.cc

It will print this if non-ASCII characters are found (and return the number of found non-ASCII characters):

Now checking file mySourceFile.h
line: 53 column: 26 nonascii °
line: 54 column: 27 nonascii €

or will print this (and return 0) if only ASCII characters are found:

Now checking file mySourceFile.h

We use it on large sets of files using bash and xargs as follows:

ls -1 include/*.h | xargs -d '\n' -n 1 checkAscii
ls -1 src/*.cc | xargs -d '\n' -n 1 checkAscii

Enjoy !

pixelstats trackingpixel
Posted in C++, Howtos | Leave a comment

Analyzing and manipulating settings.xml of LIBPF OPC

LIBPF OPC is the LIBPF module that adds plant control system connectivity via the Classic OPC interface (OPC DA). The module is configured using the LIBPF OPC Configurator, but the underlying configuration data are stored in an XML file called settings.xml.

To troubleshoot, sometimes you need to inspect the settings.xml file, and to do that you can use a few good tools.

The first thing you might want to do is looking at which groups are defined. To do that, I use the standard UNIX utilities grep and sed. These are available from any UNIX shell, but also on Windows if you installed Git for Windows (just open the Git bash shell). The command to list the defined groups is:

grep '<group' settings.xml | sed 's/.* id="\([^"]*\)" .*$/\1/g'

If you want to detect duplicates, you can pipe the output of the sed command to sort:

grep '<group' settings.xml | sed 's/.* id="\([^"]*\)" .*$/\1/g' | sort

But for more complex analyses, grep and sed soon hit their limits. The good news is that we can exploit the structure of XML, and use some more XML-specific tool, such as one for running XQuery queries in Nokia Qt SDK, called xmlpatterns.

If you create a XQuery file called groups_id.xq with this code:

<groups>
  {doc("settings.xml")//group[@id=$id]}
</groups>

then this command will filter your settings.xml and print only the group with id equal to R14AR4:

xmlpatterns -param id="R14AR4" group_id.xq

This other XQuery “program” (save as input_tag.xq) will print all input variables linked to a specific LIBPF variable, defined within groups that are enabled:

<inputs>
  {doc("settings.xml")//group[@enabled="true"]/input[TAG=$tag]}
</inputs>

One example of use for the input_tag XQuery is finding the input variables linked to the LIBPF variable S01.Tphase.mdot:

xmlpatterns -param tag="S01.Tphase.mdot" input_tag.xq

You can find more hints on XQuery here.

pixelstats trackingpixel
Posted in Uncategorized | Leave a comment

Step-by-step guide: adding new files to a LIBPF project

Reference documentation is great, but often boring to read and not helpful to solve specific problems that arise in day-to-day use. To tackle these, a step-by-step guide can be helpful.

Here is one such step-by-step guide for adding a new header/source file pair to LIBPF. This applies to the standard Microsoft Visual Studio 2008 project file (kernel.vcproj) shipped with the LIBPF SDK.

  • Add the header file to the project, selecting the command Add → New element from the context menu (RMB click) on the Header files folder of the project:
  • make sure you select Header file (.h), change the name, and make sure you put it in the include path (so that the compiler will find it):
  • In the new, empty header file, copy-paste the standard skeleton for a LIBPF header file:
    /** @file newfile.h
    @brief Contains .....
    @author (C) Copyright 2012 .....
    */
    
    #ifndef LIBPF_NEWFILE_H
    #define LIBPF_NEWFILE_H
    
    // SYSTEM INCLUDES
    
    // PROJECT INCLUDES
    
    // LOCAL INCLUDES
    
    // FORWARD REFERENCES
    
    // code ....
    
    #endif // LIBPF_NEWFILE_H
  • Do a search and replace on the file to change the file name from newfile to whatever your file name is;
  • Do a search and replace on the file to change the header guard from LIBPF_NEWFILE_H to LIBPF_YOURFILENAME_H;
  • Add the source file to the project, selecting the command Add → New element from the context menu (RMB click) on the Source files folder of the project:
  • make sure you select C++ file (.cpp), change the name, and make sure you put in the src path where source files belong:
  • Copy-paste the standard skeleton for a LIBPF source file:
    /** @file newfile.cc
    @brief Contains .....
    
    @author (C) Copyright 2012 .....
    */
    
    // SYSTEM INCLUDES
    
    // PROJECT INCLUDES
    
    // LOCAL INCLUDES
    #include "newfile.h"
    
    // FORWARD REFERENCES
    
    // code ....
  • Finally, do a search and replace on the file to change the file name from newfile to whatever your file name is.
pixelstats trackingpixel
Posted in C++, Howtos, Uncategorized | Leave a comment