v1.0_RC1

Aggregate v1.0 Release Candidate 1 (RC1)

1. further improvements to filters
2. csv file export now applies the display filter active at the moment the user presses the Export button.
3. restructure external services to make it easier to add new publishers
4. restartable cursors for paging through large sets of submissions (implemented at the datastore layer, but not yet up into the UI).
5. allow aggregate username to be designated as a superUser (not exposed in Installer; partial support in UI).
6. updated to GWT 2.4 and AppEngine SDK 1.5.4
7. remove Briefcase applet (functionality will be replaced by the Briefcase application).
8. add large dataset (310,000 record) tests for restartable cursor.

Beginning with Release Candidate 1, there will be a data migration path across all further upgrades; we don't expect any further incompatible schema changes. However, in the event that there is an incompatible schema change, you will be able to migrate your data to
a newer ODK Aggregate 1.0 instance via a Briefcase application that
can download submissions from ODK Aggregate 1.0 RC1 and onward and
upload those submissions into future ODK Aggregate releases.

An initial release of the Briefcase application will be available soon.

Upgrading

To upgrade from Beta rev.4, follow the instructions below.

For all other Alpha or Beta releases, you must either use a new appspot instance or delete ALL data in your appspot instance before
deploying the RC1. If you are using MySQL or PostgreSQL, you should start with an empty database.

On GAE

You don't need to do anything. It should just work.

On MySQL or PostgreSQL

The persistent results tables that hold the exported csv and kml files have changed (adding support for emitting csv files filtered by the active filter).

1. Stop Tomcat
2. In your database's administration tool, connect to the database and:
   * drop table _persistent_results;
   * drop table _persistent_result_file_bin;
   * drop table _persistent_result_file_ref;
   * drop table _persistent_result_file_blb;
3. copy the new WAR to the /webapps directory.
4. Start Tomcat

v1.0_beta_4: Fix for issue 311. Ill-formed form definitions cause infinite refres…

Aggregate v1.0 Beta rev.4

1. split management tab into 2 tabs (Management and Site Admin).
2. hide tabs that the user doesn't have permission to access.
3. add help tooltips and documentation pop-ups.
4. single-click hide/show of metadata in UI.
5. delete individual submissions through the UI.
6. make explicit that string fields are truncated to 255 characters.
7. allow upload of forms that do not have data types defined for all fields (with suitable warnings).
8. add odktables features.
9. added a new metadata field, _MARKED_AS_COMPLETE_DATE that tracks the date the submission was last marked as complete.
10. fix issue 311 (on Google Code) that causes infinite redraw loop if a form definition is corrupted.
11. fixed several bugs related to incomplete uploaded submissions -- submissions with only some of their media attachments present.
12. gain a global lock during instance start-up to ensure that only one instance might be changing the configuration during that time.
13. improve GAE query and task locking logic; introduce datastore settle-time logic, remove read-after-write attempts and inserted retries in key places to improve real-world performance.
14. extend timeouts for FusionTables to minimize publishing failures.
15. refactor MiscTasks, PersistentResults and FormInfo to no longer themselves be form and submission structures.
16. rework treatment of metadata so that it is more like other data.
17. clear session cookies more completely when logging out.

Upgrading

Changes to the database schema necessitate either using a clean appspot instance or deleting ALL data in your appspot instance before
deploying the beta 4. If you are using MySQL or PostgreSQL, you should start with an empty database.

v1.0_beta_3_utf-8: Add isComplete() metadata. Fix submissionDate metadata. Add settle t…

Aggregate v1.0 Beta rev.3 utf-8

This fixes 4 issues:

1. utf-8 characters are not displaying or downloading properly from a Google AppEngine server. (Issue 286 (on Google Code))
2. filtering on metadata fields does not work. (Issue 299 (on Google Code))
3. filtering does not work with date fields or select1 fields. (Issue 299 (on Google Code))
4. insert a 3 second delay in order for the datastore to settle before streaming newly-submitted data records to external services. TBD: This may not be enough time when GAE datastore is having performance issues.

This update also exposes the "meta-is-complete" metadata on a submission. If a submission with a media attachment (e.g., image, audio recording or video capture) is uploaded but the attachment is not, the submission will be marked as not complete (false). Only completed submissions are published to external services.

Upgrading

If you are running Beta rev.3, you should just be able to deploy this on top of that installation. If UTF-8 characters do not display properly in your form definition, you will need to delete the form (and submitted data) then upload the form anew.

Otherwise, if you are not running Beta rev.3, then follow the steps required for upgrading to Beta rev.3 prior to moving to Beta rev.3 utf-8.

v1.0_beta_3

Aggregate v1.0 Beta rev.3

Upgrading

To upgrade from earlier Alpha or Beta releases, you must...

On GAE

1. If running builds prior to Beta 2 on Google AppEngine cloud services, any forms with Decimal data will need to be deleted and reloaded. Decimal data had been represented as strings and is now represented properly as double-precision numbers in Google AppEngine. Note: while Briefcase can be used to download data, there is currently no way to upload your downloaded data into the beta 3.
2. Disable writes to the datastore (via Application Settings / Disable Datastore Writes )
3. Deploy to GAE.
4. Delete all the records in these kinds (using Datastore Viewer):
   * opendatakit._granted_authority_hierarchy
   * opendatakit._registered_users
   * opendatakit._user_granted_authority
5. Disable your application (via Application Settings / Disable Application )
6. Enable writes,
7. Enable your application,

On MySQL or PostgreSQL

1. Stop Tomcat
2. In your database's administration tool, connect to the database and:
   * drop table _granted_authority_hierarchy;
   * drop table _registered_users;
   * drop table _user_granted_authority;
3. copy the new WAR to the /webapps directory.
4. Start Tomcat

Upgrade Impacts

The primary impact is the loss of all registered users and their privileges (what the manual steps above do). Beta 3 changes the user configuration. Registered users are now either gmail accounts or ODK Aggregate usernames. ODK Aggregate now supports only anonymous access and/or registered users.

Known Issues

• issue 265 (on Google Code) -- IE 8 doesn't refresh the submissions page on adding or deleting forms.
• issue 261 (on Google Code) -- logins can time out while sitting on permissions page, causing unsaved changes to be lost.
• issue 231 (on Google Code) -- large submission sets may cause UI to time out.

Changelist

• recognize OpenRosa meta/instanceID tag and use it if present. Generalize to use the first meta/instanceID as the unique ID for a submission (don't require a namespace).
• change-password now uses https when available (JSOP).
• digest auth (ODK Aggregate logins) now have nonce timeout set to 30 minutes.
• better logging of submissions and reasons for failures (for GAE diagnosis, since ODK Collect doesn't necessarily report this)
• disable or hide UI functionality if user does not have required permissions for that functionality.
• update to AppEngine SDK 1.5.1
• update installer screens; https automatic on GAE deployment.
• installer now deletes its temp directory
• update security model (extensive changes)
• hide password when deploying to Appspot (issue 222 (on Google Code))
• fix launching of scripts and display of README on windows (issue 223 (on Google Code))
• show the number of media files associated with a form (issue 224 (on Google Code))
• fix crash on MacOS when some fonts are present (issue 236 (on Google Code))
• fix lockouts when changing super-user (issue 242 (on Google Code))
• hide forms that are queued for deletion from the UI (issue 255 (on Google Code))
• remove the old UI (servlet) pages.
• remove old applets
• fix operations behind proxies (issue 212 (on Google Code))
• fix lingering UTF-8 character display issues (issue 218 (on Google Code), 215)
• fix new form and upload submissions links (issue 243 (on Google Code); 257) and IE page display and upload submissions bug.
• fix PostgreSQL create-database scripting errors. (issue 256 (on Google Code))
• changes to improve client-side javascript operations, especially the refresh timer operations.
• changes to poll for configuration changes so that instances running in multi-server farms all eventually synchronize their security config and forms list.
• changes to get working (depressingly...) minimal unit tests for gae, mysql, postgresql under maven.

v1.0_beta_2

Aggregate v1.0 Beta rev.2

• fixes for Tomcat operations (missing images, styles)
• updates to tabs, stylesheets and layout of new UI
• configuration wizard improvements (Issue 222 (on Google Code), 223)
• Postgresql install works (issue 227 (on Google Code))
• updated to AppEngine SDK 1.5.0.1
• add Tomcat image preview bean
• simplified permissions tab
• change Google AppEngine decimal value storage to use double.

Limitations:

• will only display 1000 records
• will only export 1000 records to csv and kml files
• publishing broke as of Oct 2011.

To work around the publishing failures and the 1000-record limitation, use the Briefcase applet to download the data to a local csv file. Access the Briefcase applet through the URL:

https://your-application-id-here.appspot.com/Briefcase

v1.0_beta

Aggregate v1.0 Beta

Useful URLs during the install process:

Google AppEngine

Apache Tomcat 6

PostgreSQL 9.0

MySQL 5.1 Community Edition

Installer Special Instructions

You must have the Java runtime environment (JRE) installed. To download:

• Browse to: http://www.oracle.com/technetwork/java/javase/downloads/index.html
• Choose the Download JRE button.
• Follow the instructions for installing the JRE.

Installer Known Issues

• Can only select install folder; can't create it.

Aggregate 1.0 Beta Known Issues

• Security and Multiple-Running-Instance Concerns -- see AggregateSecurity

Aggregate 1.0 Beta Feature Requests

• Need indication of whether a form has associated media files