Purpose
XML input/output options for loading the database and providing upgrade paths is a core feature need for uPortal 3. This page is currently a place to gather ideas about requirements, technologies and design for such a system in uP3. Eventually this page will outline the actual implementation and architecture details of XML import/export in uP3.
I love robots.
Requirements
Database Interaction
- The system should use the standard DAO interface and implementations for moving data in and out of the uP3 database. This ensures that changes to the database structure won't break the XML import/export feature.
Object Re-use
- If possible the system should use the existing transfer and domain objects with the goal of keeping XML import/export specific data objects to a minimum. This ensures minimal synchronization will be needed when changes to the domain/transfer objects occur.
Migration Path
- The system should allow for upgrades, ie: export data from uPortal 3.1.2 and import it into uPortal 3.2.0.
Execution
- The APIs should be runnable both from the command line without a running portal available and from the web UI.
Document Scope
- The ability to import/export a specific object, groups of objects or the entire database should be available.
Formatting
- The XML format should be easily human readable and writable.
Validation / Editor Support
- It would be preferable if DTDs or XSDs could be created (by the tool or by hand) to aid in the editing of these files by standard XML tools.
Object Model Versioning
- It would be preferable if an object model version number could be included to ensure the XML will be parsed correctly by the current version of the import tool.
Object References
- A way to reference other objects in the same XML file, other XML files or already in the system is needed.
DAO Implementation Agnostic
- Database specific keys should not be included in the XML output and should not be needed in the input XML.
Affected Domains
- The import/export functionality should work for all portal domain objects including; users, portlet domain objects (6 different objects), groups and layouts.
Technologies
- XStream
- There are a number of tutorials for XStream on the site. See for example the persistence tutorial
- TSS on XStream
- Modello
- Cernunnos
- Page discussing the use of Cernunnos for this purpose
- Working Example (Please take a look!)
- JAXB
Architecture
- Options for Upgrade requirement (req 3)
- Use XSL to migrate data from an XML format specific to version A to the XML format specific to version B. This could be done internally to the import/export code if version numbers are included in the XML document.
- Add fName like (user defined unique key) to objects that need to be referenced so db specific identifiers are not used.
- Import/Export specific objects will likely be needed to allow fName references to be used instead of direct or IObjectId based references.
Comments (11)
Mar 08, 2007
Jason Shao says:
What do other containers do? Is there something we can steal? I disagree with r...What do other containers do? Is there something we can steal?
I disagree with req. 2. I want the XML documents to be very stable – between uP3 versions, and ideally even to uP2, or other portal containers. If uP3 APIs change, I explicitely don't want the XML document to change, unless some basic concept related to publishing portlets is different.
Same with 8 – what I really want is a document specification, something where the documents are versioned separately from the tools used to create/edit them.
We should probably add extensible too – built in support for some kind of <parameter> element or something like that to ensure that portlet/channel specific settings can be embedded in a standard manner.
Mar 08, 2007
Eric Dalquist says:
What about changes to the actual object model. Say a new field is added to the P...What about changes to the actual object model. Say a new field is added to the PortletDefinition, the XML would need some change to allow that field to be populated. I can clarify 2 to talk about versioning the XML separately from the framework itself (and object model version per say).
Remember this is also more than just portlets, it would cover users, layouts, groups and any other data stored in the database.
Mar 09, 2007
Jason Shao says:
I know we're talking a bunch of DSLstyle XML dialects for different data within ...I know we're talking a bunch of DSL-style XML dialects for different data within the uPortal schema. I still think that for long run stability tying the XML directly to the object model is a mistake. Witness all the pain in JAX-RPC land where SOAP endpoints end up being very brittle, as even small changes to code can cause incompatibilites in the service.
I see the XML documents as being public contracts, and thus it seems worth doing the extra work to explicitly map between the public contract and the implementation.
Mar 09, 2007
Eric Dalquist says:
I do see the perceived danger of directly mappings the XML to the domain objects...I do see the perceived danger of directly mappings the XML to the domain objects, we don't want these XML files to change very often.
Lets say we create an object tree to isolate the XML mappings from the actual domain objects. Then a new property is added to a domain object for a new feature. What do you propose is done so that new feature is available for configuration via the XML files.
Perhaps the statement that needs to be made is that the transfer object model will be treated as a public API and only changed if absolutely necessary. With that in mind I'm not seeing how we make the XML never change and still deal with adding features.
Versioning the object model and using that version number to to apply XSLT seems like an easy way to deal with changes (which would very likely be minor). If a transform is created for every change in the object model (which would be infrequent) upgrading from version A of the object model to version D of the object model is easy, just run the 3 XSLs and then the importer.
BTW, Jira and Confluence do very low level import/export. They actually use the Hibernate database to XML tools and have a custom tool to go back the other way.
Mar 10, 2007
Elliot Metsger says:
BTW, Jira and Confluence do very low level import/export. They actually use the ...I think that the exports from Jira and Confluence are different in that they are not a public contrac, and they include database keys.
Do we need to consider the XML documents as a public contract? My initial gut feeling says yes, but if the uP3 tools are solid can we say no?
For example, Jira and Confluence import and export just work. I've gone in and munged the XML data once or twice to get it to do what I need, but I wouldn't consider their import/export XML a public contract.
Mar 10, 2007
Elliot Metsger says:
Perhaps the statement that needs to be made is that the transfer object model wi...I agree with this. The OM should be treated as a public API. I'd go a little further and suggest that whatever services are layered on top of the xfer OM used to perform import/export should probably also be treated as an API.
Mar 10, 2007
Elliot Metsger says:
I see the XML documents as being public contracts, and thus it seems worth doing...So do we need to view the XML import/export of uP3 as a public contract?
I'd argue that SOAP endpoints (WSDL) and document structure of import/export are two different things. WSDL by definition is a contract.
We can decide if the uP3 import/export is a contract or not. What are the pros and cons?
Mar 11, 2007
Jason Shao says:
I think I'm looking at this in the broader context of migration as well. If the ...I think I'm looking at this in the broader context of migration as well. If the XML documents used for import/export are fairly stable, it seems reasonable to re-use them, allowing migration tools to target the (hopefully simpler than say DB representation) XML documents as an output of some migration process.
So... migrating from uP2 - uP3 --> produce the appropriate XML import.
Same goes for moving from/to Liferay, Jetspeed, etc.
Mar 11, 2007
Eric Dalquist says:
I agree that this is one of the core goals of the XML documents. That said wanti...I agree that this is one of the core goals of the XML documents. That said wanting a stable document doesn't mean new features can't be added. I'd like to hear your view on addressing changes to the OM. As much as it will be a public API, it can and will change in the future. How do we create these XML document specifications without allowing for future change?
Mar 12, 2007
Jason Shao says:
I agree that the XML documents can/should change and evolve, but I guess I'm adv...I agree that the XML documents can/should change and evolve, but I guess I'm advocating that the evolution of that document format be a consciously designed process, and not the by-product of API changes. At the same time, what I'd ideally see in a feature like this I think is less than the Confluence/JIRA all or nothing approach. It'd be great to export a single channel/config or group definition, etc. and be able to import that into a completely seperate portal instance without worrying about colliding keys, or other instance specific bits.
I guess I'm almost wondering if we want a separate XML->Object or Object->ExportAsXML factory. Some piece of code that explicitely handles translating between XML definitions, and the appropriate object scheme (and persisting?) Seems like it might be a cleaner way to isolate things like handling identity, equality, etc. Could also enable default value substitution, or make it clearer when code changes impact the public api – since the factory/translation piece would break.
More code though – still haven't fully grasped things in my head. Thoughts?
Mar 16, 2007
Andrew Petro says:
Pasted from a dreaded offlist IM conversation I had on this topic: I suspect I'...Pasted from a dreaded off-list IM conversation I had on this topic:
I suspect I'm with Jason on his last comment that a desirable thing here is domain-specific XML importer and XML exporter classes (possibly that's one class per domain, no reason to go totally profligate with classes, but I expect it's two interfaces and so could be a class specific to each) with WSDL and, ideally, admin UI bindings. The XML should be defined by formal schemas that are versioned and can change with the lovely result that exporting the XML from 3.1 and importing it into 3.2 means something.
I suspect the piecewise bit could be handled with the WSDL binding requiring correct and complete XML but the UI binding able to prompt for missing information or broken references.
The XML is the data view on the domain, the potentially human-friendly XML explanation. Java is Java. Java objects have not just data, but behavior. Java handles versioning in its specific way.
I just can't use old versions of a Java class when the API has changed. Contrastingly, the Java domain might change in all sorts of interesting ways but the XML representation wouldn't necessarily change, and even if a new version of that domain were invented to better sync, the old version could still be supported.
"Version 1.3 of the uP3 XML schema for portlet type declaration" hits the mark.
If there's an implemented-in-Java PorletTypeXmlImporter, it can look at the declared version of the XML, look at the data, and sanity check, translate using Java, translate explicitly. If it happens to be true that an XSL is the easiest way to do this in a specific case, great, it can delegate to XSL, but I think we're learning in uP2 that XSL isn't the answer to everything. In any case, the importer can explicitly look at what it's doing and sanity check it, along the lines of what the LegacyPersonAttributeDaoImpl does or what the channel publishing XML importer does (or ought to do? Haven't looked closely at that lately).