Purpose
XML input/output options for loading the database and providing upgrade paths is a core feature need for uPortal 3. This page is currently a place to gather ideas about requirements, technologies and design for such a system in uP3. Eventually this page will outline the actual implementation and architecture details of XML import/export in uP3.
I love robots.
Requirements
Database Interaction
- The system should use the standard DAO interface and implementations for moving data in and out of the uP3 database. This ensures that changes to the database structure won't break the XML import/export feature.
Object Re-use
- If possible the system should use the existing transfer and domain objects with the goal of keeping XML import/export specific data objects to a minimum. This ensures minimal synchronization will be needed when changes to the domain/transfer objects occur.
Migration Path
- The system should allow for upgrades, ie: export data from uPortal 3.1.2 and import it into uPortal 3.2.0.
Execution
- The APIs should be runnable both from the command line without a running portal available and from the web UI.
Document Scope
- The ability to import/export a specific object, groups of objects or the entire database should be available.
Formatting
- The XML format should be easily human readable and writable.
Validation / Editor Support
- It would be preferable if DTDs or XSDs could be created (by the tool or by hand) to aid in the editing of these files by standard XML tools.
Object Model Versioning
- It would be preferable if an object model version number could be included to ensure the XML will be parsed correctly by the current version of the import tool.
Object References
- A way to reference other objects in the same XML file, other XML files or already in the system is needed.
DAO Implementation Agnostic
- Database specific keys should not be included in the XML output and should not be needed in the input XML.
Affected Domains
- The import/export functionality should work for all portal domain objects including; users, portlet domain objects (6 different objects), groups and layouts.
Technologies
Architecture
- Options for Upgrade requirement (req 3)
- Use XSL to migrate data from an XML format specific to version A to the XML format specific to version B. This could be done internally to the import/export code if version numbers are included in the XML document.
- Add fName like (user defined unique key) to objects that need to be referenced so db specific identifiers are not used.
- Import/Export specific objects will likely be needed to allow fName references to be used instead of direct or IObjectId based references.
What do other containers do? Is there something we can steal?
I disagree with req. 2. I want the XML documents to be very stable – between uP3 versions, and ideally even to uP2, or other portal containers. If uP3 APIs change, I explicitely don't want the XML document to change, unless some basic concept related to publishing portlets is different.
Same with 8 – what I really want is a document specification, something where the documents are versioned separately from the tools used to create/edit them.
We should probably add extensible too – built in support for some kind of <parameter> element or something like that to ensure that portlet/channel specific settings can be embedded in a standard manner.
What about changes to the actual object model. Say a new field is added to the PortletDefinition, the XML would need some change to allow that field to be populated. I can clarify 2 to talk about versioning the XML separately from the framework itself (and object model version per say).
Remember this is also more than just portlets, it would cover users, layouts, groups and any other data stored in the database.
I know we're talking a bunch of DSL-style XML dialects for different data within the uPortal schema. I still think that for long run stability tying the XML directly to the object model is a mistake. Witness all the pain in JAX-RPC land where SOAP endpoints end up being very brittle, as even small changes to code can cause incompatibilites in the service.
I see the XML documents as being public contracts, and thus it seems worth doing the extra work to explicitly map between the public contract and the implementation.
I do see the perceived danger of directly mappings the XML to the domain objects, we don't want these XML files to change very often.
Lets say we create an object tree to isolate the XML mappings from the actual domain objects. Then a new property is added to a domain object for a new feature. What do you propose is done so that new feature is available for configuration via the XML files.
Perhaps the statement that needs to be made is that the transfer object model will be treated as a public API and only changed if absolutely necessary. With that in mind I'm not seeing how we make the XML never change and still deal with adding features.
Versioning the object model and using that version number to to apply XSLT seems like an easy way to deal with changes (which would very likely be minor). If a transform is created for every change in the object model (which would be infrequent) upgrading from version A of the object model to version D of the object model is easy, just run the 3 XSLs and then the importer.
BTW, Jira and Confluence do very low level import/export. They actually use the Hibernate database to XML tools and have a custom tool to go back the other way.
So do we need to view the XML import/export of uP3 as a public contract?
I'd argue that SOAP endpoints (WSDL) and document structure of import/export are two different things. WSDL by definition is a contract.
We can decide if the uP3 import/export is a contract or not. What are the pros and cons?