Admin: Configuration Management

Background

Spoon manages configuration files in $HOME/.kettle for repositories.xml, .spoonrc, etc.; and in $HOME/.pentaho/metastore for NamedCluster, Default Run Configuration, etc. This is also true for webSpoon: user configurations are managed in these folders of the server. These folders are owned by the user who runs the Tomcat; thus, every end-user accessing the same server shares the same configuration. To make webSpoon truly multi-user ready, these configuration files have to be managed separately for each user.

Scope

There are other things that have to be considered in terms of configuration management; namely, third-party JDBC drivers, plugins downloaded from marketplace, etc. These things would require some architectural changes; hence, I'd like to put them aside and focus on $HOME/.kettle and $HOME/.pentaho/metastore here.

How should they be managed?

The most straight-forward way is just assigning different directories to different users as Spoon. Alternatively, user settings can be stored in a database like many other webapps. Whether files or a database, it is desirable that they are not locked-in to a single webSpoon instance, but shared by multiple instances when scaling-out. The other option is a browser cookie: The RAP Deverlopers' Guide says user settings can be stored in a browser cookie, which expires in 3 months though (hard-coded in here).

The table below shows what each role is required to do.

Method	Developer	Admin-user	End-user
Files	Implement user authentication	Manage users and directories	N/A
Database	Implement user authentication, Change how to save/load configs	Manage users and database	N/A
Cookie	Change how to save/load configs	N/A	Configure per browser

Current status

0.7.1.11 has taken the first approach "Files". As a result, a part of configuration files in $HOME/.kettle/users/$username (see the table below) and all in $HOME/.pentaho/metastore/users/$username are dedicated to $username only if user authentication is enabled like described in here. If not, the same configurations direct under $HOME/.kettle and $HOME/.pentaho/metastore are used for anyone accessing it.

Item	Description	As of 0.8.0.13	Desired	Comments
kettle.properties	Main PDI properties file; contains global variables for low-level PDI settings	Shared	<--	This affects all users; hence, should be managed only by admin.
shared.xml	Shared objects file	Dedicated	<--
db.cache	The database cache for metadata	Dedicated	<--
repositories.xml	Connection details for PDI database or solution repositories	Dedicated	<--
.spoonrc	User interface settings, including the last opened transformation/job	Dedicated	<--
.languageChoice	Default language for the PDI client tool	Shared	Dedicated	It takes more engineering effort to change, but one global setting may be just fine.
xulSettings.properties	Not documented	Dedicated	<--

The directory structures are as below.

$HOME/.kettle/
├── .languageChoice
├── kettle.properties
└── users
    ├── user1
    │   ├── .spoonrc
    │   ├── data
    │   │   └── Untitled.ktr
    │   ├── db.cache-7.1.0.0-12
    │   ├── repositories.xml
    │   ├── shared.xml
    │   ├── shared.xml.backup
    │   └── xulSettings.properties
    └── user2
        ├── .spoonrc
        └── data

As can be seen in the directory tree, data folder is experimentally created under $HOME/.kettle/users/$username/data. This folder is meant to store Kettle files and prevents them from being dispersed in the server.

$HOME/.pentaho/
├── caches
│   ├── ehcache
│   └── libfonts2
├── classic-engine
│   ├── system
│   └── user
└── users
    ├── user1
    │   └── metastore
    │       └── pentaho
    │           ├── Default\ Run\ Configuration
    │           │   └── Configuration\ 2.xml
    │           ├── NamedCluster
    │           │   └── test.xml
    │           └── Spark\ Run\ Configuration
    │               └── Configuration\ 3.xml
    └── user2
        └── metastore
            └── pentaho
                └── Default\ Run\ Configuration
                    └── Configuration\ 1.xml