Projects/Modularity/Reference Data

Reference Data tools required in Modularity

Overview

One of the coolest features of Openbravo ERP Modularity is the ability to include reference data in a module. For further detail about what Reference Data see the Modularity functional spec. The goal is to provide tools for developers to export data from their instances to include them in their modules and to automate the import process when installing/applying modules. It will be possible to share not only new functionalities developed on top of Openbravo ERP but also any kind of data needed to run those functionalities. It will highly improve the user experience because many configuration and data loading tasks can be automated and hidden to the user. Examples of reference data are taxes for a particular country, product categories used in a particular Industry or security configuration templates.

Types of reference data

In Openbravo ERP 2.40 and earlier releases there are three types of particular reference data supported: translations, chart of accounts, and import data through files (products, business partners, etc.). From 2.50 release and ahead the concept of reference data is generalized and any data in the instance can be exported in a module and imported when installing/applying this module.

Modularity will support translations, chart of accounts and standard reference data. Import data is not supported since there is no value in supporting it because it can be done through standard reference data.

Translations

Openbravo ERP has a set of tools to manage the translation process. Go to the translation management project for further details.

In the modularity project no new tools are created to manage the translation process but it will allow developers to include a language pack in a module and to automatically apply that language pack to the instance when the module is installed. It means that the language in that module will be set as System Language in the instance (if it was not) and the translations applied so after installing it the user will be allowed to log in the System in that language. When updating these modules the translation information will be automatically updated. When uninstalling these modules the translation information will be deleted but System Languages will not be modified to avoid crossed effects. It means that after installing and uninstalling a translation module the instance might be modified by adding a new System Language to it.

To improve maintainability and search capabilities of translation modules the only content allowed within them is translation files for just one language. Translation modules will be marked as "isTranslationModule" and will have just one dependency to the module that it is translating for. The native language of the translation module is of course the language used for translations. All this information will be publically stored in the Central Repository. By this mean it will be easy to look for translation modules to a particular language for a particular module. The name, description and help of a translation module will be equal (but translated) to the name, description and help of the module it translates. By this mean users will be allowed to search for modules in their own language. If the user decides to install a translation module the module it translates will be also installed due to dependencies.

Chart of Accounts (CoA)

Openbravo ERP provides a mechanism to load the chart of accounts from a .csv file at client and/or organization level during the Initial Client Setup and the Initial Organization Setup.

In the modularity project no new tools are created to create the .csv file but it will allow developers to include one (just one) Chart of Accounts file (.csv) in a module. These modules will be marked as "hasChartOfAccounts". When these modules are installed in an instance those Chart of Accounts are stored in the System to be used during the Initial Client Setup and/or Initial Organization Setup.

In the Initial Client/Organization Setup processes the user will be allowed to choose which ones of those modules are applied to the Client or Organization. The same module can not be applied at both levels in the same instance because it would lead to data redundancy. When a module that has a Chart of Accounts is applied to a Client/Organization the result is fully equivalent as if the user had chosen the .csv file included in that module.

Modules including Chart of Accounts can be updated and uninstalled. When updated they will be shown in the Client/Organization module manager as available to update. When updating them it will be fully equivalent as if the user reloads the Chart of Accounts from Openbravo ERP proper window. When uninstalled from the instance it will have no effect since it is not possible to delete Client/Organization information. The user will be able to install other Chart of Accounts and update Client/organization properly.

Standard Reference Data

Starting from release 2.50 and thanks to the new Data Access Layer (see DAL project page) Openbravo ERP provides a mechanism to export any data in the instance to xml files and to import data from those xml files. A module can include a set of xml files exported through this mechanism. These data will be imported in the instance that installs the module at installation time (System information), Initial Client Setup (Client information) or Initial Organization Setup (Organization information).

The following sections will explain in detail how the export tool works and how data is imported in the instance.

Data Sets

In 2.40 and earlier releases DBSourceManager had a Java class to define the dataset of application dictionary members: what are the tables where application dictionary metadata is stored and conditions for rows (SQL filters) because not all records on these tables are members of the application dictionary metadata. New requirements are needed to have a more accurate definition of the AD:

• to define what columns on these tables are members. Sometimes it is easier to define which columns remove, for example to remove audit info
• to define conditions for columns to remove calculated values, for example to remove the name, description and help from ad_fields that are centrally maintained and this info is hold in ad_elements

The concept of datasets is also applicable to reference data. To improve the maintenance experience of modules including reference data it is required to persist within the module the definition of the reference data set so the task to export module information can be automated. The definition of the reference data set is fully equivalent to the one used to define the AD, consisting in a set of tables, conditions, columns and column-conditions.

Data sets are defined in a new data structure in the application dictionary that is operated through a standard Window to edit that information. The data structure to define datasets is made by three tables:

• DataSet, with the following columns: Value, Name, Description, Module and DataAccessLevel. Parameters (TBD: for example, modules for AD data set).
• DataSet_Table, with the following columns: DataSet, Table, fullBusinessObject (boolean), includeAllColumns (boolean) and whereClause (HQL)
• DataSet_Column, with the following columns: DataSet_Table, Column, isExcluded (boolean) and conditionClause (Java expression)

DataSet

Data sets have a name and a description to describe the content of the data set. The value is used to get a dataSet object from the factory provided by DAL (eg. DBSourceManager gets the AD dataSet by this mean). A data set is owned by a module in the same way that all Application Dictionary components. DataAccessLevel is used to define how to import that information. Available values are:

• System: data will be imported at module installation time at System level without any user interaction.
• Client: data will be imported at Initial Client Setup if the user chooses the module where the DataSet is included.
• Organization: data will be imported at Initial Organization Setup if the user chooses the module where the DataSet is included.
• Client/Organization: data will be imported at Initial Client Setup or Initial Organization Setup if the user chooses the module where the DataSet is included. The module can not be applied at both levels at the same time because it would lead to data redundancy. So if the module is applied to a Client it will not be available to apply in its Organizations and if the module is applied in an Organization it will not be available to apply in its Client.

System/Client and All values are not allowed.

DataSet_Table

A data set can have one or many tables from the ones registered in the AD_Table. For each of them developers can decide to include only records in that table or export the full business object using the check fullBusinessObject. A full business object is a record including all its one-to-many relationships as defined in the AD through the isParent attribute of a column. An example of a full business object is a product with its vendors, prices, etc. A complete description of business objects is provided in the DAL project. Developers can also define for each table the columns that are included in the dataset. They can include all columns using the includeAllColumns check and then remove some of them in the column definition or only include the ones that are explicitly defined in the column definition. The whereClause is a HQL expression to filter the rows that are included in the DataSet. Details on this expression will be provided in the DAL project.

DataSet_Column

For each table in a data set developers can decide what columns to include from the ones registered in the AD for that table. They can exclude columns using the isExcluded check if they have marked the table as "include all columns". Typically audit info will be removed from the dataset.

Transient Columns

Optionally they can also exclude a column through a isTransientCondition defined at Column level in the Application Dictionary. This is an expression written in Java that access the values of other columns (properties) in the row and returns a boolean value. If the expression is false the column for that particular row will be marked as transient. Details on the syntax of this expression will be provided in the DAL project. Another way to is to exclude a column is mark it as isTransient in the AD, for columns that always be transtient, e.g. AD_Table.Sql_Record_Identifier

AD data sets and DBSourceManager

There are three data sets needed to describe the Application Dictionary metadata in a way that DBSourceManager is able to create and update the database based on this information:

• Application Dictionary metadata (AD). Eg. AD_Windows, AD_Table, etc.
• Application Dictionary referenced data (RD). Eg. AD_Client, AD_User
• Application Dictionary metadata in configuration scripts (CS). It's a subset of AD with only the elements that can be modified in a configuration script.

These three datasets are managed in a different way that the generic ones used for reference data. The three of them are accessed by DBSourceManager through a DataSet Java object created by a factory provided by DAL and requested by DBSourceManager using their values.

When creating the database DBSourceManager will insert in a clean database all data defined in AD and RD. When updating the database only AD data set is used, so RD is loaded when the database is created but then it is not kept in sync with the source. When calculating the configuration script for a particular Industry Template only the CS data set is used.

Be aware that AD data set is managed by DBSourceManager in a different way that other data sets are managed by export/import tools. The key difference is that DBSourceManager aims to keep this data set in sync with the source. It means that after an update DBSourceManager guarantees that the data set in the instance and the data set in xml files are identical (so it may delete records if required). It will be done by comparing both data sets, and for each difference executing a statement in the database to apply that change (if a new row an insert, if a modified property an update, if a row has been removed a delete). It's also important to notice that DBSourceManager guarantees that after the update both data sets are identical. It means if the data set definition change from one version to a new one, DBSourceManager will use the new one to generate the current data set in the database.

Export tool

DAL will provide a mechanism to export the information described in a data set from the database in the developer instance to xml files. The export tool will take care of all the details described in the data set. In particular columns (properties) where their conditionClause returns false will be marked as "transient". All rows (business objects) referenced from the rows (business objects) included in the data set will be also exported (and the ones referenced by these ones, and so on) so they can be imported without the risk of foreign key constraints errors. The only exception to this is for Client, Organization and audit information since they are properly managed by the import tool. The structure and syntax of those xml files are described in the DAL project.

The xml files will be stored in a folder (referenceData) within the module package. When the .obx file of a module is generated it will include those xml files so they will be available for importing when the module is installed.

From the data set header the user will be able to execute the export process at any time and the xml files will be properly updated. The process that generates the .obx file will optionally execute the export process of all data sets included in the module to guarantee that information is up to date.

Import tool

DAL will provide a mechanism to import a set of xml files obtained using the Export tool. From the Openbravo ERP application menu the user will be able to import data by choosing a xml file from his/her local file system. The user can also decide if data will be overwritten or ignored if some business objects included in the xml files are already present in the database (they will be identified by the UUID).

If a row (business object) to be imported references to a row (business object) that is not registered the import tool will import it during the process to avoid foreign key constraint errors. Be aware that the xml file includes as well all referenced information. If the referenced row (business object) is already registered the import process will ignore this info (so it will not be updated).

If a validation fails when importing data the business object will be discarded and a proper message will be shown to the user.

The import tool will manage the audit information: CreatedBy and UpdatedBy columns will be set to the user who is performing the import and Created and Updated will be set to system date/time. If the record is only updated because is was already registered then only UpdatedBy and Updated columns are updated.

The Import tool will take care of the "level" where the information is imported:

• If the dataAccessLevel of the data set is System information will be imported at System level (Client 0, Organization 0). UUID's of business objects are kept as they were exported.
• If the dataAccessLevel of the data set is Client, Organization or Client/Organization the import tool will ask the user for the Client and the Organization to import those data. Of course the process will be secured so the user can only choose Clients and Organizations he/she is allowed to write. The import tool will create a copy of all records in the data set for that particular Client and Organization with a new UUID for each record. The process will store in an auxiliar table (AD_Ref_Data_Loaded) the mapping between original UUID's and the generated ones at Client/Organization level so it will be possible to know what is the original data for a record in the instance. Any reference to those records from other records in the data set will be transformed to the current one for that particular Client/Organization.

It's worth to stress that all data is imported at the same level regardless the level it had when it was exported. For example, if a data set includes a tax zone that references to a country, if that country is not present when importing it will be created at the same level as the tax zone itself (TBD).

Using standard reference data in modules

Installation time

When a user installs a module all data sets at System level will be automatically imported in the instance when the installation is applied (system is rebuilt). By default the import process will overwrite data if it is already present. If the user updates a module with a System level data set those data will be imported again overwriting previous information. The import process will never delete information so if developers wants to remove some data from the data set they have to deactivate those rows. If the user uninstalls a module with System level data nothing happens in regards of those data: they are kept in the system as they were imported. It means that after installing and uninstalling a module with System reference data the instance might be modified by adding those data to the system.

Initial Client/Organization Setup

When a user executes the Initial Client Setup he will be allowed to choose which modules with reference data to apply to the Client from the ones that have been installed in the System and contain a data set at Client or Client/Organization level. The Initial Client Setup will automatically import those xml files into the Client during the process.

When a user executes the Initial Organization Setup he will be allowed to choose which modules with reference data to apply to the Organization from the ones that have been installed in the System and contain a data set at Organization or Client/Organization level and have not been applied to its Client. The Initial Organization Setup will automatically import those xml files into the Organization during the process.

Client/Organization module management

User can as well to maintain modules applied to existing Clients and Organizations. From this window they can apply, unapply and update modules to Clients and Organizations.

Users will be allowed to choose which modules to apply from the ones that have been installed in the System and contain a data set at proper level and have not been previously applied to its Client (in case of Organization) or to an Organization in the Client (in case of Clients). It will automatically import those xml files into the proper level during the process.

Users will be allowed to choose which modules to un-apply from the ones that are applied to the Client/Organization. In this case nothing happens in regards of those data: they are kept in the system as they were imported.

Users will be allowed to choose which modules to update from the ones that are applied to the Client/Organization and a new version of these ones has been installed in the System. In this case those data will be imported again overwriting previous information. The import process will never delete information so if developers want to remove some data from the data set they have to deactivate those rows.

Closed Discussion Items

ICI: how to manage dependencies for translation modules? It is stated that dependencies are set at Major Version level. It means that all Minor Versions within a Major Version of a Module are compatible for the module that depends on it. For translations modules it does not work because it is likely that labels or descriptions in modules might be changed in Minor Versions and by so forcing Translations Modules to be updated to properly translate them. -> It makes no sense to stop the upgrade of a module because its translation is not complete. It means that the user will be allowed to update their modules although their translations are not complete, so we will get a UI that is partially translated. it fits perfectly with Modularity principles in regard of dependencies.