uPortal Developers Meeting March 27-28, 2006. Rutgers University
Attendees
Dan Ellentuck Columbia
Andrew Johnston Columbia [Tuesday only]
John Fereira Cornell [Monday only]
Michael Oltz Cornell
Jonathan Markow JA-SIG
Faizan Ahmed Rutgers
Tom McKeon Rutgers
Dan Mindler Rutgers
Jason Shao Rutgers
William G. Thompson, Jr. Rutgers
Mark R. Boyd Sungard-SCT
Jan Nielsen Sungard-SCT
B. Collier Jones UMBC
David Grimwood Unicon
Cris J. Holdorph Unicon
Michael Ivanov Unicon
Peter Kharchenko Unicon
John Lewis Unicon
Andrew Petro Unicon
Adam Rybicki Unicon
Andrew Wills Unicon
Ted Kantner Unisys
Eric Dalquist UW-Madison
Susan Bramhall Yale
Bruce Raker (via video conference) University of Delaware
Brad Johnson (via video conference, 1st day only) Texas Tech
Jim Helwig (via video conference Monday morning) UW-Madison
John Hare (via video conference Monday morning) UW-Madison
Two groups from the University of Kansas (and KUMC) were also videoconferencing Monday morning.
The videoconferencing worked very well Monday morning. Five different groups dialed in via ISDN and viewed the meeting. There was a good bit of trouble Monday afternoon; the room manager said that one of the sites was apparently looping the data back to us, which caused problems with the system. My (Michael Oltz's) speculation is that when putting their systems on standby for lunch break monday, one of the sites pushed the wrong button. Once that site, whichever it was, gave up trying to reconnect, those still trying were able to get in. On Tuesday, only U Del participated.
Common uPortal-specific and some other abbreviations
A "layout manager" takes care of acquiring the content for the user's layout pages and putting them together for display. At present there are three different internal architectures being used for handling this.
SLM -- Simple Layout Manager, the earliest implemented of the three. Does not have a concept of "fragments" (centrally managed portions of a user's layout).
ALM -- Aggregated Layout Manager, introduced in uPortal 2.2. Allows mixing tabs which are centrally created and modified ("fragments") with tabs that the user creates and arranges as desired. Has "integrated modes," i.e. the mode of using the portal, and the mode of editing a user's layout, are both on the same page.
DLM -- Distributed Layout Manager, similar to ALM but with more powerful capabilities. Written at Sungard-SCT for their Luminis product and offered back to the open source code. Included in uPortal 2.5, but not the default. Does not yet have integrated modes, although this is planned.
GAP -- Groups and Permissions, part of the uPortal base code written at Columbia U. This allows the consideration of uPortal users as members of groups, and assigning them permission to do various things (authorization) depending on what groups they are in.
PAGS -- Person Attributes Group Service, a "thin" way of using (primarily) LDAP attributes to determine authorization. This is one of the several existing ways that GAP can determine group membership.
CAS -- Central Authentication Service, originally developed at Yale, now an open source project under JA-SIG
HyperContent -- A website-content-management system (CMS) originally developed at Columbia University, now an open source project under JA-SIG
WSRP -- a specification for how to use Web Services (data bundled in XML, requested and returned in a standardized way) to get the contents of a portlet from a remote host.
JSR-168 -- The existing Java community standard for writing "portlets," individual avenues of information within a portal.
JSR-286 -- A near-future Java community standard for writing "portlets," enhancing JSR-168.
FishEye -- web-based software to inspect and manage a source repository.
Confluence -- a particular wiki software
Jira -- issue-tracking software closely related to Confluence
MONDAY
Introduction -- Eric Dalquist UW-Madison
Bill Thompson Assoc Dir for Enterprise Systems and Services Rutgers U. (host for the meeting)
In his brief welcome and introduction, Bill mentioned The Rise of "Worse is Better" by Richard Gabriel
an approach to software development -- 'the New Jersey principle' get something out and get it to be simple which encourages participation, even if it's not perfect.
JA-SIG Web Presence
John Fereira, Cornell University
The main uportal web site is maintained by HyperContent, and is mostly very static.
A lot of work has been done recently on the web site.
They're trying to make a facebook, so that visitors would be able to see the people filling various roles in the uPortal community.
There had been no form or link to send email to anybody, so they did that.
There is now a photo gallery; intended for photos from meetings etc.
There has been some talk about using a common layout, analogous to what the JA-SIG CAS site has.
There is now a site developer.ja-sig.org; it has a link to the FishEye (repository management) software, and other things of developer interest.
As of Friday, March 24, 2006, the site is now at the Princeton clearinghouse machine (used to be at the University of Delaware).
There is now an RSS feed to post uPortal news.
The most dynamic part of the site is when someone wants to make a new release of something, and put a link to where to download it.
John has not yet made docs for how to use HyperContent to add content to this site.
He migrated most of the existing documentation over to the wiki.
The thing that is needed next, is to get a list of who is implementing and who is in production.
[a bug was brought up re the move to Princeton: if you go to "uportal.org" you don't get the same thing as "www.uportal.org"]
John took on the Ja-sig conferences site as well. It is a webapp written in Java, and needs some overhauling.
There should be more of a single signon solution for all the JA-SIG stuff (which is in several sites).
Bill T: There has been some discussion of this in the CAS project.
Eric D: We are trying to get Confluence and Jira sync'ed at least.
John: The HyperContent site is already using the same userid database as the uPortal Jira.
Jason: Someone needs to find the time to do this. e.g. there is no clear owner of the Clearinghouse uPortal instance.
Susan B: Has the developer part of the site been officially announced? Didn't know about it.
There needs to be better linkage to it.
Dan Mindler: Tried to structure stuff under 'developer' starting about 6 months ago; now it may be time to make it better known.
Jason: There is a Google Group about JA-SIG; actually it was started by people working on the JA-SIG website to discuss improvements to it. Let's make a part of web presence that is more about marketing/why should you use uPortal.
http://groups.google.com/group/jasig-webpresence
Please suggest more things to do with the web site.
JA-SIG email services
Eric Dalquist & Jason Shao
Jim Helwig at UW Madison said that they could offer better email facilities than what we already have at U New Mexico
Eric: We are set up to switch over to this. We could have the address be "@ja-sig.org" We could have a larger number of targeted lists. We can delegate authorities so more than one person can create and manage email lists. What are people looking for, what suggestions do you have?
Bill T: Sounds great to do this. Especially having a 'security' mailing list.
John F: If we create new mailing lists, the new ones would not be archived at UNM; so you'd have to migrate all the archives to UW so they can be found in one place.
Jason S: We are beginning discussions speculating how easy the migration would be. We were going to try migrating it into Confluence, but haven't done it yet. "I'm hopeful this [migrating email archives to Confluence] is a problem that's already been solved."
Eric: What are good listnames and alias names? Do we want aliases for project leads?
Jonathan M: It makes sense to split out the projects.
Andrew P: I'd like to see 'jasig-developers' for the metaissues that affect every project's development.
John F: Should there be abstractions like "info@ja-sig.org"? Should there be individual people, like "patty_goetz@ja-sig.org"?
Jason: For now, let's limit aliases to roles and services, rather than handing out individuals. Revisit that later.
Susan: There is a lot of overlap so it's nice to have the collective mailing lists; but my imperssion is that HyperContent seems to be a separate community at present. [i.e. there seems to be less crossover of discussions and people than between other parts]
Eric: At first we will split out the mailing lists to be more specific. More importantly this move [to a new hosting site] will let us be more flexible for the future.
John: Should we migrate the Google Group over to the new mail server?
Jonathan: The Clearinghouse mailing list should be over there too. More specifically a problem contact point.
Jason: Are there lists that should be public, and some private? 'security' should be private. sometimes people toss around pathnames and install places.
Someone gave examples of what we might do:
Aliases: security, webmaster, postmaster
Mailing lists: uportal-user, uportal-dev, cas-user, cas-dev, jasig-dev
2.4.x status
Dan Mindler, Rutgers
We had a security release 2.4.3.1 a minor release in December, 2.4.4. to run under JDK 1.5. just a few bugfixes. Rutgers has been using 2.4.x for a couple years, now migrating to 2.5.x this summer. We want someone else to manage 2.4.x.
Susan: Why did you not go to 2.5.x earlier?
Bill T: The differences are minor internally, but we had a major feature release for local stuff so we didn't have time to move over.
Dan E: What is your packaging philosophy; what are the remaining issues?
Dan M: Mostly there have been backfitting features from 2.5.x and the HEAD.
Eric: This is interesting re our new strategy for managing projects. After Ken left, we split it up and put each person on a branch. Now how do we as a community deal with someone no longer wanting to be the point person for a branch? This is the first such case.
Bill T: The community did a lot of work on the forward release strategy. We need to clarify expectations for end-of-life.
Andrew: The only reason it might be "EOL" is that nobody wants to do it. But if someone later comes along to do it, it can be revived.
It was recommended that we should get more clarity about the status of releases and going forward; there is confusion among user community about "where should I be if I'm in the 2.x universe?" and there is confusion about the status of 3.x. The developer community needs to be clearer as to what these releases are.
Dan M: Rutgers' focus for 2.4.x was on stability and performance rather than new features.
Faizan: Was the chart from the Baltimore conference on the wall helpful? [The 'wave' let each site subjectively indicate how far along it was in adoption and acceptance of uPortal]
John F: We should have something on the web site that keeps track of this. Could we have a release-specific support list?
Bill T: Because Ken was doing everything, what the roles were was not clear. Maybe there should be someone in charge of keeping track of the downloads page, and of who is running which release.
Eric: Unlike on other open source sites, [e.g. Apache], we haven't been saying, "We recommend you use version x.y.z".
Jonathan M: We don't want the lack of such information to limit adoption.
Faizan: Maybe have an 'adoption' session at the conference?
Faizan: For us 2.4.x is really stable, the fact that we are leaving it doesn't mean it's no good, we don't want to give that impression.
Eric: But there's an advantage to having everybody on the same release. We want to be careful what we say to potential adoptees
Jonathan M: We need to say what we recommend and how we position the other branches.
Mike Z: We should have an official policy that we will support X releases back but if someone wants to work on something earlier, that's okay.
Bill T: Create a community where the path of least resistance is to move forward. Rather say "The board or community leaders will endeavor to find release engineers for X releases back."
Jonathan M: I don't know what the correct number of releases is, but that's the right message.
Jason: One reason that people use open source is that commercial vendors keep pushing them forward in releases.
Jan N: For example Tomcat, there's 3.x, 4.x, 5.0 and 5.5. The driving force is whether there are enough people using a given release that there can be support for each other.
Dan M: We aren't anticipating any new minor releases for 2.4.x.
Andrew: The difference between 2.4.x and 2.5. is rather modest. Since then we've done a lot more thinking about releases. The upgrade path there could have been a lot smoother; the release number might not have needed to be bumped up to 2.5.x; but there are important differences in support of portlets.
MORNING BREAK
uPortal 2.5.x status
Andrew Petro
"The most important thing about 2.5 status is that Rutgers is moving to it. You want to be on the release that Rutgers is on." (this was stated somewhat tongue-in-cheek)
2.5.2: There are a lot of small bug fixes; CAS client support is in the box; concurrency fixes especially for portlet support.
The theme for 2.5.2 was bug fixes.
We need a theme to focus on for 2.5.3 and who wants to help work on it.
Dan: There are some significant performance improvements for GAP, might be ready.
Andrew: Now big schools are trying to deploy 2.5.x; up to now it has been adopted by serious but smaller schools.
Jamison Watkins: Wanted to talk about WSRP support.
Jonathan M: That has come up in a couple of different contexts recently. 1) Lots of interest in the UK; possibly some available resources to help. 2) UDel began a group focused on PeopleSoft-uPortal interoperability, particularly via WSRP. They demo'ed a WSRP producer out of the next People Tools; it didn't get consumed properly in uPortal. Lots of interest in UK and USA for Oracle portlets. An Oracle techie offered to help us.
Andrew: A pattern has emerged: resources are also the infrastructure to test this stuff. It's a huge asset to have an example of such a WSRP that we can't render, so we can address the issue.
It seems to be an anti-pattern when these type of conversations go off the list and other developers don't hear about them.
Jason: Is there a Jira issue about [the problem rendering the Oracle producer]? If we make sure there is, that can be a focal point of the discussion.
Who is on 2.5.? Yale, SCT; and Rutgers is going in that direction. Unicon is going to do some load and performance testing on it soon.
Eric: There are bugs in DLM under high load. DLM + PAGS + PersonDirectory makes it hit LDAP an awful lot unless you use caching. Otherwise it took 30 sec to 1 minute for initial login.
Rutgers is moving to the 2.5 branch; they have been handling the packaging of releases for 2.4. Perhaps it would be advantageous for them to do packaging for 2.5 now, in which case Andrew could turn more of his attention to other things. But unless and until some such thing happens Andrew will continue to do it.
uPortal 2.6.x status
Eric Dalquist and Jan Nielsen
A release is determined by configuration file changes, schema changes; or how much would it affect local modifications. e.g.: U Wisconsin has an extended stats recorder interface; in one of the maintenance releases of 2.5.x there were some changes to the stats recorder internals which made it harder for UWisc to upgrade.
Bill T: There is a release strategy document that the community worked on and released. People who make local mods should let others know what they are doing.
Eric: I wanted to talk about I18N, DLM changes, other features, and what is the 2.6.x release strategy.
Jan: If the community wants what we (Sungard-SCT) are doing, there are a lot of configuration and source changes. We won't be in a position to contribute those changes until we finish our project. So we don't want to hold up what 2.6 is, if you want it it would be later in this year. Who should be the release manager? Is SCT-HE appropriate for this? Our position is we don't think it's appropriate to be in that role so we are looking for someone.
Andrew: There is this bundle of changes; good I18N sounds like something for a particular release. What else would be beyond 2.5 that would break things? I haven't heard of anything?
Susan: What about the things being discussed tomorrow morning?
Dan E: Possibly relevant
Eric: Scott Battaglia -- proposes that we take the stats recorder we have now and use an event-based model instead, using Spring.
-- GAP improvements
-- moving PersonDirectory into separate jars
-- new stats recorder
-- I18N
if that doesn't happen until this fall or early next year, is that a problem?
Jason: If we're going to another minor release, maybe we should do some other housecleaning, removing deprecated API's etc.
Andrew: Could we embrace the Spring web MVC, and use it much more pervasively?
Bill T: Maybe the previous list of features wouldn't get people moved to 2.6. We need a vision-building roadmap so we have criteria for what features to choose.
Jan: Is there a page on the wiki for 2.6 proposed features?
Bill T: There's a discussion at a more sweeping level, what does higher ed need in an enterprise portal.
John F: The Spring MVC would make a shorter roadmap from 2.x to 3.x.
Susan: Where are we on a roadmap to a roadmap to get from 2.x to 3.x? Ways to get the infrastructure in place to make the transition easier.
Jason: There is a session tomorrow about the uPortal development process.
Someone: There are several initiatives going simultaneously, and we need to clarify the message that we give. There's 2.x work, there's 3.x work, and there is discussion about long term requirements. We need to tie these together for the June conference and talk about where we want to wind up.
Eric: This is why the session tomorrow is an hour and a half. There's lots to talk about.
Jonathan M: Let the community know what the requirements process is. And the relationship of 3.0 to the overall goals needs to be clarified.
Bill T: Integration with enterprise systems is a significant issue for us.
LUNCH BREAK
uPortal Internationalization
Jan Nielsen
Uploaded slides to wiki. What Sungard-SCT has done on their version of uPortal. [they have a commercial product which embeds uPortal]
goals --
produce a single binary across all specific locales.
externalize every locale specific string into resource bundles.
let it look like the core product does today.
let user community localize to other locales.
deployment environment --
Luminis 4 runs in Java 5 JVM, J2EE 1.3 compliance.
basic requirements --
sysadmin defines default locale of the JVM
resources that are available in that locale will be displayed to the end user.
ISO 639 2 letter language code and ISO 3166 2 letter country code.
User specified locales -- if the sysadmin has allowed other locales, and there's more than one, the user can choose among those locales for their personal default. (the locale switches immediately)
Fixed layouts -- we support only left to right, top to bottom (so no Arabic or Chinese). We will address those later and these will have framework implications.
Customer localization requirement -- an end customer will get the stock Luminis product. If the customer wants to they can create the locale. Some users have interest in locales that don't have a big enough business case for the vendor to do it (such as Welsh).
Use the canonical JDK localization constructs to do most of this.
Scope --
The entire presentation layer.
Tactically -- messages that are in files beyond the presentation layer e.g. log messages.
We don't expect to have a system log that's in anything other than English.
But users may want a system administration log that's in their own language.
1. externalization of resources -- move every locale specific string and put them into resource bundles. May be a Java file, may be a JSP file, may be XSL, etc. The string is put back in by parameterized replacement at run time. For example, they use JSTL for JSP. At the Java layer, it's java.text.message.MessageFormat. Their resources support primitive formatting: paragraph and line breaks, italics, and bold.
2. To put Unicode into .properties files you must use ASCII-encoded Unicode (\u0000 formatted characters). This is not directly human-readable, but the JDK provides the "native2ascii" tool to convert a Unicode encoded file into an ASCII-encoded Unicode file.
3. Luminis uses unique resource names, e.g. the fully qualified class name, what kind of resource it is, and a name for the resource. This makes the source code look less attractive to developers. So use resource keys that are as descriptive as possible but still usable. They mark which resources are error messages ".failure." and which are log messages ".log." Of course there are other kinds of messages.
Resource format:
.properties files use 8859-1 character encoding as said earlier, and when Unicode characters are needed they use \u0000 format. They support macros in the form
${substitute_something_here}
In their case they have over 2000 properties, most of which are in a single .properties file.
How do you name a resource that more than one source file uses? You can make a generic name and map it with a macro to the source-specific name.
They have an API that gets asked for a resource, and goes through the appropriate resolution process to get the result.
They use one resource file per source file. Many other projects just have one resource file for the whole project.
We need some way in the UI to show where the I18N has actually been done.
Jan gave examples of what Java, JSP, XML, Javascript, etc. looks like when using parameterized resources. JavaScript and XSL had more issues in how to do them than in the others.
"resman" is their tool for managing resources.
Jan put two documents on the meeting wiki, one with the overview of how it works and what it looks like in the uPortal source, and another with more technical information.
John Lewis: How would this transition to uPortal 3.x?
Jan: We haven't spent any time in uPortal 3 as yet. The general comments probably apply; Spring uses simlar constructs to do I18N. It might not be as big an issue.
Eric: We haven't looked at I18N for uPortal 3. There was some discussion about various technologies. the answer was "Whatever somebody decides to use is what we will use".
Peter K: We have looked at one aspect of this for uPortal 3, in the preferences.
Jan: A user selects one locale; the user does not choose several different locales in order of preference. We didn't want to replace the functionality of ResourceBundle, which is what the previous model essentially did.
Peter K: [Another approach to implementing the ordered list of locales would be] If we know what translations are available in the system, we could then go through a list of preferences, and pick just one, before we do any looking up of specific resources.
DLM Status Update
Mark Boyd, Sungard SCT
ChannelStaticData
-- persists in user's layout
-- honors user overridable settings
Fragment Manager (there is one)
Two phases of work remaining for DLM
1st
replaces dlm.xml
still edit fragment layouts via separate accounts
pushed/subscribed fragment
disable/enable
audience determinations using groups manager
localizable
granular permissions
migration (dlm.xml to up_dlm_fragments)
2nd phase
in-session fragment layout editing
integrated modes
Parameter processing pipe
history
UserInstance refactoring [there was too much ALM specific code in it]
processUserLayoutParameters()
LM-specific processLayoutParameters()
I/M Initial work (Tension)
LM must remain oblivious to the structure and theme
But Structure rules for "AddChannel" very simple
add a layer in front of the LayoutManager to look at the URL and possibly react to it differently
package:
org.jasig.portal.layout.dlm.processing
[he showed highlights of the API, see his slides]
DLM is Spring configured
There is a fixed processors list
OptionalProcessors Map
OptionalProcessor selectable via request parameter uP_dlmPrc=key, persists until replaced or asserts isFinished()
Can participate in Parameter Processing and/or SAX processing
He gave two simple examples of Parameter processors
Pluggable
Structure/Theme specific processing
Flexible
customer request parameter processing
injection of structure and theme stylesheet parameters ...
Channel parameters (fall 05)
Fragment manager (post Luminis 4.0)
ParemeterProcessingPipe (this week)
I/M (post Luminis 4.0+)
Jason -- Is Fragment Manager in the HEAD?
Mark -- No, it depends on Jan's I18N work, and we have to decide whether that's going into the shared code.
AFTERNOON BREAK
DLM as default when?
Susan Bramhall
When Susan asked this question, she thought it was tough to make the switch between ALM and DLM. Now it's clear to her it's less work than that.
Andrew: DLM was not the default in 2.5. firstly, because it was brand new at 2.5.0, and secondly, because only some of the example channels were available in the DLM default layouts. [So, it was not known how many bugs remained and it was not in a condition to be evaluated by potential adoptees]
Eric: Someone should write a layout publishing tool before we make DLM the default.
Jason: Rutgers has a command-line tool that injects SLM layouts into the database.
Dan Mindler: We thought it had been contributed but it was written for Rutgers by Unicon. We will contribute it.
Dan M: What impact would making the DLM the default, have on people who are already using another layout? Any?
Susan: Is ALM being enhanced at all?
Eric: No. Some people are looking at moving to DLM.
Jason: Migration questions are not necessarily relevant to whether making DLM the default is a good idea.
Jason: We could ship the simplest default, or we could ship with the layout that shows off what people could do.
Andrew: We need to be clearer about what we expect people to do. Don't run SLM because DLM is a superset of the features; don't run ALM because no work is being done on it.
Someone: We want new users to start with DLM which is best going forward and not be faced with an unnecessary upgrade.
Andrew Willis: Cal Poly is migrating from 2.1.4 or 2.1.6 to 2.5.1 or 2.5.2; from SLM to DLM. Hoping to go live in April; Unicon implemented for them, publishing and distributing of DLM's.
Susan: So we could add layout publishing tool to the 2.5.x branch, then make DLM default later. People would like to have the publishing tool at the same time, not have to wait for it.
Eric: ALM has a command-line publishing tool but DLM does not. So making DLM default without one would be a step backward.
Dan M: There are three kinds of users with respect to this question. 1) Newbies, migration irrelevant. 2) People who want to migrate. 3) People who don't want to migrate, but want to upgrade code base.
Andrew: I recommend we make DLM the default in 2.5.3.
It was agreed that this makes sense as long as there is a layout publishing tool for it.
Chris Holdorph: We were doing performance testing; we recommended 2.5 and DLM but we had two errors UP-1224 infinite recursion; and sometimes things would just disappear from the layout, other not in Jira yet. So the user chose to go with SLM instead.
This problem is Jira issue UP-1224. Mark Boyd said he would look at it. Needs to be resolved before DLM is made to be default.
Project Channel/Theme/Layout Change Management
Jason Shao
It should not be so hard to change themes etc. for local sites. Should not have to tinker so much to get it to work.
For example, SLM and DLM theme transformations have some commonality, can part of their flow be merged?
He showed what Rutgers' default theme looks like; most of it comes out of the CSS scripts rather than in the HTML. (The site's policy is no guest access; therefore visitors won't be able to go look at it "live")
[I missed some remarks that Jason made about possible enhancements.]
Collier's work on myVT (at Virginia Tech) on a more agile theme has generated lots of favorable comment; perhaps we could find a way to flexibly fit that work back into the base.
As there was a little extra time, two things not on the original agenda were brought up.
Dan Mindler briefly raised an issue: We need a channel deprecation strategy. There's no easy way to retire channels.
Andrew: Packaging up the U-Wisconsin code. UWisc developed some custom portlets that other sites are interested in. We have incremental progress on getting that out in open source. The Web Proxy portlet is available, and is in CVS as a project. There are customizations to the portlet adapter to allow scoped caching. And it keeps statistics relevant to portlets rendering.
Eric: There is also multithreaded rendering. One user gets one request in the pipeline at a time. If you are proxying large binary images e.g. it would otherwise hang up everything else. There isn't much code change to fix this, but it does change how things are done. This needs a lot of discussion because of all the U-Wisconsin mods it has the most implications on how people do things.
A "PortalControlStructures" object. Each user gets one of these, and it has a lot of objects in it. If you start having multiple requests coming in at once you have only one place to pass things around. With the multithreaded rendering mod, there are static threadlocals in the PCS. The class that fires off worker threads digs in to the PCS and gets the appropriate stuff out of there for the rendering thread. it does not seem to generate any memory leaks. But this is a fundamental change required for how portlets work. So it is needed before people could use the U.Wisc code.
Mark B: Why couldn't you do this in the download worker instead?
Eric: Because we don't know what we're going to get when we get the URL request.
Someone asked Eric about the 'extended personalization interface for working with DLM.' There is some code to do this when editing fragments. But it is tab-specific and the status of this is not clear.
Eric: On the wiki there is a list of all the customizations that UWisc has done that they would eventually like to get out.
http://www.ja-sig.org/wiki/display/JSG/UW-Madison+Contributions
As a layout admin he can 'pretend' to be a template user in order to easily edit the fragments.
TUESDAY
PersonDirectory update
Eric Dalquist
This refers to the attempt to separate the PersonDirectory code into its own jar.
All done except for one piece of code which should be done this week.
One open issue is LDAP connectivity. It's been using the uPortal infrastructure code to get at LDAP. Now that it's going to be separate it needs to get at LDAP some other way. What about using LdapTemplate (SourceForge).
Andrew: Scott Battaglia, when working on CAS, faced the same issue. Jason added, Scott originally used Spring LDAP support but decided to use LdapTemplate.
PersonDirectory will be pulled out from 2.6 into a separate jar; this change will be backported to 2.5 if it doesn't break anything.
Jason: Why backport it?
Eric: To reduce the number of places we would have to maintain the code going forward.
Groups and Permissions ("GAP") update
Dan Ellentuck
The goals for the spinoff of GAP code:
-- Create an independent library from GAP framework to support uPortal 3
-- Make GAP more straightforwardly configurable
-- Make components of this system more easily replaceable
components
-- common infrastructure
-- group services
-- groups core
-- permissions
-- full
-- uportal2 api
Done so far:
core services
common services
local group service
filesystem groups service
unit tests (95% done)
build process
configurable using Spring
Instead of properties file there is now a Spring bean definition
Remaining work:
Migrate the PAGS group service (dependent on spinoff of PersonDirectory)
Migrate Entity Name Finder service
Apply permissions enhancements (Keith, Mark?)
Migrate JitLDAP group service (Matthew Ling)
Documentation (especially for implementers)
Build process controlled by Maven
VFS group service?
Address performance issues
uPortal 3 issues? There are some constants (IGroupConstants) in uPortal 2; Peter doesn't know where they moved in uPortal 3; Peter made an interface and added those constants back. (Everyone group, etc) should these be in the framework side? Dan's first approach was to define these in the Spring bean definition.
Peter: The uP3 project has a way to expose the GAP to portlets. Probably we should distribute that for up2
Dan: VFS [Jakarta Commons Virtual File System] -- this makes many different data access mechanisms look like a file system. Dan wants to have a groups service that works on top of that.
Dan: Should we use the spun-off GAP libraries for uP2? Whether and when?
Eric: As long as the replacement is functionally equivalent of what's there, I don't see that it would be any problem for 2.6 or 2.7. The question reduces to same as PersonDirectory -- could we retrofit it to 2.5 as well without causing problems, so we wouldn't have to maintain two different projects?
Susan: If there's no functional improvement, but we would still have to reconfigure, then it's only an inconvenience. So why do it?
Andrew: We could make some adapter that still supports the old config files, then it wouldn't be so bad.
Eric: The contract is if we remain in 2.5.x we should not require reconfiguring anything, and we should not lose or break any API's.
Susan: One benefit is that the site is being prepared for a later update that won't be as difficult because of this.
Jason: The Eclipse project has a document about how they manage their public API's [may be relevant].
Jan: There are some tools from Maven (jdiff) that help you manage the public contract and make sure that later releases maintain API compatibility.
Dan: Do we feel we have a quorum to make the decision [to use the jar in 2.x] or do we need broader community input to make this decision?
Andrew: "I think we have a very talented functional owner of this component," and to a large extent, what Dan decides is best, is what we should do. If Dan has serious reservations, we should not.
Dan: What I think the question is, at what point do we inflict the pain? I think I'm over familiar with the GAP and the Spring files, so I'm not the best judge of how much pain it would be to convert.
(Dan continues)
Performance issues. He's done some profiling of GAP on 2.5. It's been influenced by Columbia's move to DLM. There are three bottlenecks:
"Deep" methods. These allow asking the question, is person a recursive member of some group? These are heavily used by DLM. This can 'hydrate' (instantiate the entire object graph of) the entire group system. This particular problem does not happen with PAGS which does not 'know its members'. But others like local or filesystem do know. Dan thinks this happens because of an inappropriate algorithm for how it evaluates this. It's a straightforward fix to short circuit this search; however it's an implementation detail which kind of changes behavior. I certainly want to put this into the HEAD and the jar code, but what about 2.5?
Then there is excessive object creation in large group systems. The fix would be to have groups in general cache group keys rather than actually acquire an object. This would require changing the internal API for the group store (group DAO). which would then be inappropriate for 2.5. However, this has a huge impact on performance for large group systems.
The next one is the use of javax.naming names, which is rather inefficient. This is also a potentially significant change.
Consider how PAGS, PersonDirectory, and uPortal deal with IPersons. Should there be a registry for only one IPerson per user, so that only one instance gets passed around? This would be much more efficient. PAGS is not provided an IPerson from the session, because we didn't want to pass around an IPerson with a securitycontext in it; Ken thought there should be a 'restricted' IPerson.
Eric: If we change the internals so that we don't load the whole group tree in some circumstances, would that change the results of the query?
Dan: No it should not.
Jan: The public contract is the public API and that the results of calling it are the same. If the changes result in changing the functional output of that API, what you do instead is to offer a slightly different API with the new results, and leave the old one alone.
Dan: Keep discussing this. Let me know offline.
Jan: Do you have a feeling for the amount of dependencies you have today?
Dan: No, I don't. I'm not aware of a large number of group connectors out there, but I'm sure there are.
uPortal3 update
Eric Dalquist
At the moment, the biggest effort is going into making sure that the portlet spec is properly supported. [see earlier remarks about the Oracle/PeopleSoft portlet]
Layout
Group management
PersonDirectory
In up3 we have the portlet IChannel adaptor. Peter will talk about this after the break.
The other change is moving from Spring 1.x to Spring 2.0 which has portlet MVC code in a released version. And use the domain-specific configuration capabilities.
Peter: The WSRP producer DAO's are being changed to Hibernate.
Also upgraded Groups to Spring 2 as well.
Doing a lot of bug fixes.
Peter: There aren't any features in the first beta to help conversion from 2.x; the first beta will be mostly to just try for new adopters.
MORNING BREAK
uPortal 3 DLM Integration
Eric Dalquist
uPortal 3 has only SLM. We want to integrate DLM. One issue is the persistence layer. When we port to up3 we could take the opportunity to revise that. Do we store XML? Do we store CLOBs?
Mark Boyd does not have time to do it. the uP3 team would enjoy an offer of someone's time to take this on.
Peter: What are the main steps that it would require?
Mark: Layout Manager
persistence layer
configuration loader.
management channel.
Mark was asked to send a list of components out on the dev list.
Andrew: We have a requirements process. Let's figure out what makes DLM so attractive and valuable, and why people are going toward it, and publish that.
Jason: Yes, on the one side we have a feature set, but on the other side we should give reasons why the feature set is there.
Peter: There is a more specific question -- there are some features that are really key, and others that could be modified. If we have a list of motivations, we will know what is most crucial not to damage or change in the port. It would be a good point to also say whether there are specific features that need to be changed if we have time. Would a wiki page do this? Like "ALM gap analysis" page that already exists.
Portlet IChannel Adapter for uP3
Peter Kharchenko
On the wiki page for uP3, there is some architectural documentation.
The architecture is a mirror image of what we have in uPortal 2 now (an IChannel which can contain a portlet and drive the life cycle). In uP3, we have "ChannelAdapterPortlet" which can contain an IChannel and drive its life cycle. The idea is to fake a uPortal 2 environment. We have to provide "StaticData", "RuntimeData" classes etc. which have to be modified so they work with what is available in uP3.
This uses a channel adapter similar to up2, it supports many kinds of up2 channels such as multithreaded etc. There is some work left to do, where some of the lesser classes involved in this are working exactly as they did in up2, and should be rearchitected to work internally in a up3 manner.
IPrivilegedChannels cannot be supported, because there is no equivalent of "PortalControlStructures" class. Well, you can run an IPrivilegedChannel but it can't get a PCS.
Identifying specific requirements: We don't know what code is in the various channels around the world; instead, get community agreement on certain important channels and make sure they are available in up3. For the effort involved to be worthwhile and reasonable to do, the up2 version must be either in the vanilla distribution, or unmodified in the Clearinghouse, and it must be requested for porting by many or large sites; if a site has a custom channel or custom modified channel, then that site will have to do its own porting or reimplementation in up3.
Mark B: Does up3 have a concept of preferences (structure, theme, etc.) like in up2?
Peter: It's 'attributes' which can be associated with a theme, style, etc.
Mark B: My concern is configuring DLM, how will that be done?
Important channels mentioned in the session:
UBC_Webmail
-- uses address book as a servant.
Announcements ("both versions")
Adam -- it uses servants, make sure they are supported
User Manager channel. [which is included out of the box in uportal 2.5]
Yale menu channel.
PersonAttributes
What about ICCRegistry? [Peter: would be difficult to implement]
HelpDesk channel -- very popular in Europe. it's a SourceForge project.
adminapp channel -- Peter: This should be converted to a portlet.
Bookmarks channel -- make sure old bookmarks can be converted to the new portlet version.
In general we would need to be able to migrate the tables for the various channels.
Cornell RuntimeInfo channel -- Susan said it's really valuable.
Blackboard WebCT channels
There are a couple things not yet supported:
CAR file support
IMimeResponse classes -- will compile but won't work.
Servant channel concept? [Peter -- hasn't tested it, should not be a problem], GAP, address book
Jason: Inability to upload binary data is limiting.
Peter: No technical reason, just hasn't been addressed yet.
There is not a permissions manager portlet yet, but one will be needed by beta.
Mark B: Will we be supporting the IPermissible interface?
Peter: Well we will need GAP so yes we will. We had been waiting for the 'detailed permission management' to be checked in, but Mark [above] said it's not coming; therefore we will not have to wait.
Some features of portlets will be much easier to do when JSR-286, the successor to JSR-168, comes out, providing more powerful portlets API.
uP3 is using Acegi for security.
Andrew: We need LocalConnectionContext. A channel can get a proxy ticket by way of a CASConnectionContext, so we need this capability. Let's try to get a LocalConnectionContext [???securitycontext?] from a portlet, in this week's coding session.
We should prioritize the requested channels, which ones we must have, which ones we would like/are looking for someone to do it.
Migration tools from up2 to up3
Peter Kharchenko
We are going to have a first beta release for new adopters, with no migration capability.
The final 3.0 release would come with a set of migration tools. No work has been done on them as yet.
Identify the scope today:
-- How much automation do we expect?
-- How would we implement some of these tools?
The main issue is data migration.
Most of the configuration (formerly .preferences files, to be Spring config files) will have to be done manually anyhow because the format, and what info is in them, is completely different.
We would expect the persistence layer of DLM to change, so even that would require migration.
One way to do it is to dump the data to XML and use an XSLT transform.
Another is to massage existing Layout Manager to read a single user's layout into XML then write it back out, but this would be tough.
Or, write a whole new tool that will read the database, and reassemble layout fragments and move them over.
Peter: Are there tools to extract fragments?
Mark B: There are no particular tools. Fragments are just regular layouts with a few more columns.
Layout move
User information move
Critical channels move
User preference values
Structure and theme definitions?
Peter: There is a backward compatibility feature that reads .sdf files.
Groups and permissions
Peter: The persistence layer is the same right now. No plans to modify it.
Everybody's layout after conversion should be valid; each channel will be pointed to a new equivalent or a wrapped channel.
Question: Have you [Peter] looked at the tables in the database to see if some look particularly troublesome to convert?
Peter: Not yet.
Eric: You can base some of the work by looking back at your template users and seeing how a user has modified a layout from that. This can reduce the complexity of the migration. You have to have the groups and channels and portlets migrated before you migrate the layouts. What pieces of migration require a tool, and what can be done by hand?
Peter: More conversion is required when converting FROM a channel TO a native portlet. Not so much when converting from a channel to a portlet-wrapped channel.
Jason: There should be some way to have a concept like a subscribe-ID carried forward. Some sites have data associated with a particular INSTANCE of a channel, and use the subscribeID to look up the data. It doesn't have to be the same character string but if not then you have to have a mapping and some way to decipher at runtime or to let the site do local db migration.
Mark B: Every layout today has a header folder, a footer folder, and a hidden folder. Some stuff is put in there. CharacterCachingChannelIncorporationFilter, Mark modified this so it can make those folders defunct.
Dan E: Experience having shown that few migration tools are perfect and most time is involved in handling exceptions, it would be vital to have the "before" and "after" state of the database and configurations documented.
AFTERNOON BREAK
Faisan: Sessions needed for summer conference?
We need people for "Portlet Showcase". -- a competition with audience approval and a prize. So far there's only one submission.
Most suggestions were from Andrew:
Open/Close Principle -- managing resources.
Migration from IMultithreaded, why was it deprecated and how easily can one do so.
There are a lot of new faces, not presenting yet. Brad's name kept coming up.
JetSpeed 2 -- any lessons we could learn for uPortal?
uPortal Development Process
Jason Shao [uPortal framework development space]
The document on the wiki referring to this, came out of a conversation between Rutgers and Eric during his last visit here.
-- how do you get new people involved in the uPortal development process?
-- strategic plan: How do we develop a vision for the long term for the future?
We should have a set of documents or a working definition of where the uPortal platform is going.
How do we leverage our community in terms of driving the product forward.
One goal is to reduce the frequent forking of the code. How do we communicate local changes back into the main product?
One thing to do is define what the roles are.
A deliverable is to have a document which defines what uPortal is for and where it will go (based on comments from the various user sites) in general.
Then something which details specific features which are or will be necessary.
One role is the "release engineer". Deciding when to release, ongoing checking up with people about progress on individual items, deciding what to postpone, packaging.
Another is "core developer" people who are very familiar with the framework and particular parts of the portal. They give advice, some support, and code.
Another is "institutional representatives".
Another is the general community of users, deployers.
a) Make long term goals clearer.
b) Making development easier by describing what the requirements are for each feature. So the developers don't have to put in a lot of effort gathering it themselves.
Bill T: We had discussions and we put it up for the dev list. Today is the next step: critique the contents, possibly add to them, soon put it out on the user list.
Jason: We have this 'bootstrapping' document to get the process started describing where we are and what's going on.
As we continue, these documents should become part of the development process itself.
Peter: If I wanted to participate right now [in this role- and future-defining process], I need an example of how to do so. We need a recipe for how and where to put things [not just the technical wiki editing, but sociologically and procedurally].
Jason: That's not solidified yet.
Jonathan M: Perhaps have a form or template, use it, and show an example of the result.
Peter: Don't we want more detail than in the current examples? So show the level of detail we want, then others will not hesitate to put in that much detail themselves.
Bill T: Two things I'm hearing --
a) How to participate in this process?
b) Flesh out at least one point for real.
Jonathan M: Sakai had a requirements gathering process; their template has additional attributes for each requirement. One attribute was "Is your institution willing to contribute any development effort toward this requirement?" another is "level of importance".
Perhaps we could have "value to the suggester."
Faisan: a) Do we also want to capture people's opinions about how the feature should be implemented?
b) How can we capture that a feature is no longer wanted by anybody?
Jason: a) What do we want to do and why?
b) How do we think it should work and why?
Then we could look backward and say "ah, now I see why such and such was implemented this way instead of that."
Jonathan M: One thing I'm hoping is to get more info from end users about usability and presentation. I don't think we're getting enough input about that.
Jason: Individual implementers may have "gripe lists" "I wish I could fix this but I can't justify the time to address it". If these concerns are made public, maybe people would come forward or a group would evolve that could collaborate to do it.
Bill T: If there's 20 institutions that have the same need, it's hard to envision that one of the core implementing schools would not want it.
Bill T: I think we've got a good process and framework, but it won't do us any good if the community doesn't adopt it as a practice. We need to keep up the momentum for this idea. Besides taking this for a spin, also if you are particularly interested in one area of uPortal you could jump in and coordinate or help the progress of that part of the design/futures documents.
Jason: a) Request to look it over and add topics
b) An abstract role of 'framework coordinator' -- it's very difficult to get one person to do everything, but let's see if we can find someone to solicit and coordinate people to take on the known roles, or to participate in the designing documents and to ping them to update the entries.
Our developer meetings should result in a document that defines what people have been doing in the last few months, what is coming out in the next few months, and what features are planned for the longer term.
The community wants to feel how active or vibrant a project is.
Deliverables for the conference: a) To give a report on development status
b) To take the opportunity to do requirements gathering.
Over the next six months:
Jan -- Internationalization; please read my document that I referred to during my presentation, and comment on it
Eric -- uP3 portlet support and bug fixing [PersonDirectory in up2]
[who said this, was it Mark?] Fragment Manager and Integrated Modes support
increased support for PeopleTools/WSRP
Columbia -- 2.5, performance testing.
Andrew -- 2.5.3 theme of 'performance'/stability
Jason -- XHTML theme work
Pete Boysen? -- adding drag and drop to DLM?
Layout publishing for up3
Layout publishing for up2
Getting up3 ready for new installations
Looking for a up3 specific theme.
Things we would like to see but have no resources:
DLM in uportal3
Web proxy portlet -- UW could give support and guidance to some other site.
AFTERNOON BREAK
Things people will work on during the coding session
Eric -- Finish up PersonDirectory then move on to up3
Andrew -- DLM fix and enhancement
Collier -- theme change
Andrew -- how close are we to have uP3 managed with Maven?
uP3 Code Review
Peter Kharchenko and Eric Dalquist
What do people want to look at?
Jason: How about the uP3 rendering pipeline?
Peter took us to a wiki page that describes up3 rendering.
There is a rendering class that wires up the various components then calls the various steps in the pipeline.
IStreamPipelineUpstreamComponent
IStreamPipelineDownstreamComponent
In the stream package there's RegexStringFilter
In the Sax package there's the XSLT stuff, TransformationFilter, serialization implementations as well.
Most of the time you want to use a "hybridPipeline" that wires together other pipelines. To wire two Sax strings together you have to set the connections between them.
Look in uP2_static_context.xml in the WEB-APP directory.
Go to line 167. the <bean id="hybridPipeline"> definition.
We cache the result of a particular transformation as a string, then we operate on a string to insert portal content.
first element of pipeline is the
userLayoutManagerSource -- so we have a source of the raw layout info.
transientLayoutInjector -- injects the transient folder which contains portlets referenced by functional name
structureTransformationFilter
themeTransformationFilter
sax2stringFilter -- serialization for the Sax string and is essentially a caching entry point
[from that point on in the pipeline it is a string rather than XML]
renderingInitiationFilter -- will pick out the portlet tags to kick off rendering of those portlet windows
portletContentIncorporationFilter -- puts the output of the portlets into the page.
writer -- outputs the string to the servlet.out
The cache is declarative because we do it in Spring.
One cache per object type so we can have distributed caches.
Framework provides ability to delegate cache support to somebody else.
To add cache at a particular pipeline you would have to make a class that implements ICacheFilter interface and then you don't have to worry about interaction with other caches.
The cache managers turn out to be specific to component type.
"Rendering Attributes" -- these are applied in the pipeline during the transformations
IRenderingAttributeInjector -- the implementation injects attributes into XSLT as elements.
webapp/rendering_attribute_beans.xml
If you have a layout manager that implements the uportal2 interface, then the "LegacyLayoutSourceAdapter" will make it run. You must make sure your data is in the right place and the persistence layer is properly configured.
You also have to have a class that knows how to read the stylesheet.
URLConstructionXSLTStylesheet
In the Java code or the stylesheet, you should not be writing out explicit URL syntax. Instead you should call special classes for the purpose of generating URL's.
Peter again opened the floor, what do people want to know about?
Jason: Under the portal layout package, there seems to be some classes for ALM?
Peter: That's an old commit, it's not the same ALM that's in up2. We got to a certain point then were taken off that part.
Andrew: How much other code is there that's inactive, besides the ALM?
Eric: There's probably a fair amount of inactive code (a few classes here and
there). There was some JDBC persistence code that's moved to the attic
because we did Hibernate.
Andrew: How about the Acegi integration?
Peter: webapp/security_context.xml
Acegi looks at every request and goes down through a chain of authentication providers to decide whether authentication is necessary or if you can just let them proceed.
The trick of Acegi is that we can apply aspects to the [???] to do authorization.
Jason: Status of WSRP in up3.
Peter: The producer and consumer work. Haven't tested it with PeopleSoft. We had a producer registry in the consumer, that's been rewritten in Hibernate.
The producer persistence on the producer side has been also rewritten but hasn't been checked in.
We would like to have more testing done on this feature.
Andrew: We should have the build process more automated (Maven etc.) so that people don't have to wait for someone to do "heroics" and a snapshot happens.
Jason: How do we get people interested in working on the new thing? Can't we backport something from up3 to up2. Wouldn't that get people interested in working on up3 itself?
There was some discussion about this.
Peter: There are common issues that up2 and up3 have. For example DLM has some problems. Maybe we want to improve the persistence layer. So we do this in up2 first, and then move it over to up3. He thinks that spending time importing new features back to up2 would be a lot of work and detract from the up3 development.
What things or issues facing up2 would be helpful to up3?
DLM
Groups manager
Permissions manager
GAP itself
PersonDirectory
If any of the important channels are going to be worked on extensively in up2, consider reorganizing them to be more portlet-like.
Someone: Suppose I were going to port some big commercial app to up3. What would be impediments to this?
Peter: layout manager requirements aren't going to be met
Better subscription portlet needed.
Load testing
Hibernate persistence layer still needs some work.
It is not yet known when 3.0 alpha will be out.
3.0 beta will be feature complete for new installers.
Adam: There is no plan to implement JSR-286 or WSRP2 in uP2.