Skip to content

Latest commit

 

History

History
760 lines (488 loc) · 29.3 KB

README.textile

File metadata and controls

760 lines (488 loc) · 29.3 KB
Raw

CC Rails Portal Activity Authoring, Deployment, and Reporting System

Setup

Prerequisites

Working git, ruby or jruby, and rubgems, wget
Gems: capistrano, capistrano-ext, ruby-debug (oddly, haml too, when deploying …)
TODO: understand why running capistrano requires haml…

Install

The latest setup scripts have been tested with 1.4.0RC1 of JRuby, you will need a running mysql
server and a mysql username/password with enough permissions to create and drop the
database you intend to use with RITES.

RVM, Bundler, & Gem Dependencies

Updating to a newer codebase.

Core Extensions

Setting up a Hudson CI project

GIT

In the example below we use the xproject theme and also as the name of the directory and
the prefix of the names for the databases that will be created in mysql.

If you are a committer in this repo:

git clone [email protected]:stepheneb/rigse.git xproject

If you are not a committer:

git clone git://github.com/stepheneb/rigse.git xproject

Change to the project directory:

cd xproject

Setup rvm to use Ruby 1.8.7 and a gemset named xproject

rvm use 1.8.7
rvm gemset create xproject

Make an .rvmrc file so rvm will use this ruby gemset combo automatically when you change to this directory:

echo "rvm use 1.8.7@xproject" > .rvmrc

Test out the .rvmrc file and start using the gemset

cd ..;cd xproject

Install bundler

gem install bundler

Install the rest of the necessary gems:

bundle install

Automatic setup of application settings in the config directory, for example: settings.yml and database.yml

ruby config/setup.rb -n "Cross Project Portal" -D xproject -u <dbuser> -p <password> -t xproject --states none -y -q -f

The option --states none means that only a single virtual district and school will be created.

If you leave this option out the default is to create portals instances for all the districts and schools in Massachusetts.

You can also supply a comma-delimited list of two character abbreviations for states or provinces:

--states RI,CT

The options -y -q -f to setup mean:

  • -y automatically answer ‘yes’ to accept default values
  • -q use ‘quieter’ console output
  • -f force an update of the existing settings.yml and database.yml if they exist

Setup database.

If this fails it is probably due to a bad mysql gem install, check out the RVM page on database integration for help

RAILS_ENV=production rake db:create
RAILS_ENV=production rake db:migrate:reset

Setup application resources

RAILS_ENV=production rake app:setup:new_app

Save a copy of the development database to make subsequent clean starts much quicker (bypassing rake app:setup:new_app)

  • Saves database to: db/development_data.sql
  • can be reloaded later with: rake db:load
  • note: in default setup created by config.rb above the development and production database is the same
rake db:dump

Start server and open http://localhost:3000

ruby script/server -e production

You can read this documentation at: http://localhost:3000/readme

After getting the server running it’s good to confirm that all the tests pass before changing any code.

Prepare a database for use when running the tests:

rake db:test:prepare

Run the rspec unit tests:

rake spec

Run the cucumber integration tests:

rake cucumber

All these tests should pass. If you add features make sure and add tests for these new features.

Roll your own theme:

For now the best thing to do is to copy an existing set of themes. eg:


  cp ./config/themes/<old_theme_name>/settings.sample.yml ./config/themes/<new_theme_name>
  cp -r ./themes/<old_theme_name> ./themes/<new_theme_name>
  cp -r ./public/stylesheets/themes/<old_theme_name> ./public/stylesheets/themes/<new_theme_name>
  cp -r ./public/stylesheets/sass/themes/<old_theme_name> ./public/stylesheets/sass/themes/<new_theme_name>
  # edit the assets file to package your style sheets. copy an example.
  vim ./config/asset_packages.yml
  # then pass arguments to setup.
  ruby ./config/setup.rb -t <new_theme_name>

when deploying to a new server

  1. create required directories on your server eg:
    1. /web/production/APP_NAME/shared/log
    2. /web/production/APP_NAME/shared/config
    3. /web/production/APP_NAME/shared/initializers
    4. /web/production/APP_NAME/releases
  2. put configuration files in /web/production/APP_NAME/shared/config
    1. at a minimum you need database.yml and initializers/site_key.rb
  3. modify the deploy recipies (in your local config/deploy.rb and config/deploy)
  4. deploy cap deploy ( it will fail, but it will get far enough to make some of the other things below possible)
  5. run ruby config/setup.rb on the server
  6. comment out the one line in config/initializers/rites.rb
  7. make sure config/nces_data isn’t there
  8. run RAILS_ENV=production rake rigse:setup:new_rites_app

there’s a bunch more that needs to go here

Copying the production database to a development environment

If you have ssh access to a portal instance running on a server you can get a copy of the database on
your local development instance with the following steps:

In the code below stage means a capistrano stage that identifies a remote server. For example xproject_dev.


    cap <stage> db:fetch_remote_db
    RAILS_ENV=production jruby -S rake db:load

If the codebase on your development system has moved forward you may need to run additional tasks such as:


  RAILS_ENV=production jruby -S rake db:migrate
  RAILS_ENV=production jruby -S rake rigse:setup:default_portal_resources
  RAILS_ENV=production jruby -S rake portal:setup:create_districts_and_schools_from_nces_data

The task: default_users_roles_and_portal_resources is last on that list because code changes may have added additional and necessary default model initialization.

In order for the same passwords to work you will also need to have the same keys in your local config/initializers/site_keys.rb as on the server you copied the production data from.

  cap <stage> db:fetch_remote_site_keys

Recreating a new portal instance from scratch from an existing application.


    rake db:drop:all
    git clean -fXd
    jruby config/setup.rb
    jruby -S rake gems:install
    RAILS_ENV=production jruby -S rake rigse:setup:new_rites_app

The ‘-fxd’ parameters to to git clean:

  1. d: Remove untracked directories in addition to untracked files.
  2. X: Remove only files ignored by git.
  3. f: force removal

Setting up a local JNLP Web Start servlet

You can also setup a local jnlp web start server for a development environment with less dependence on outside services.

  1. Install Jnlp Servlet and build associated WAR file with jnlp and jars

NCES District and School Tables

When a rails-portal instance is created two tables containing data for schools and districts in the US are created from data supplied by the National Center for Education Statistics.

NCES maintains a database about US districts and schools called the Common Core of Data

The rake task: portal:setup:create_districts_and_schools_from_nces_data downloads 2006 NCES CCD data files from NCES website and imports data from these data files into the following models:

  • Portal::Nces06District
  • Portal::Nces06School

Only data from states and provinces identified in the config/settings.yml for the portal instance are imported.

The NCES district and school models are used to provide data from which districts and schools actively using the portal are be created.

The Portal::Nces06District includes about 50 different fields of data for each district.

The Portal::Nces06School includes about 500 different fields of data for each school.

PDF documentation for the NCES data schemas

Testing

Testing Frameworks

Rspec, Rspec-rails

Cucumber

Webrat

Factory Girl

factory_girl allows you to quickly define prototypes for each of your models and ask for instances with properties that are important to the test at hand.

Using Nokogiri with JRuby on Mac OS X

Some of the testing frameworks depend on Nokogiri which is a Ruby html and xml parsing gem that uses the C-based libxml2 library.

When Nokogiri runs in JRuby it uses Ruby FFI to dynamically load the libxml2 shared library.

The version of libxml2 (2.6.16) included with MacOS X is old and the FFI version of Nokogiri prints this warning when it is run with this version of libxml installed:

You’re using libxml2 version 2.6.16 which is over 4 years old and has plenty of bugs. We suggest that for maximum HTML/XML parsing pleasure, you upgrade your version of libxml2 and re-install nokogiri.
If you like using libxml2 version 2.6.16, but don’t like this warning, please define the constant I_KNOW_I_AM_USING_AN_OLD_AND_BUGGY_VERSION_OF_LIBXML2 before requring nokogiri.

If you have a newer version of the libxml2 library installed with macports you can set this environmental variable:

  LD_LIBRARY_PATH=/opt/local/lib
to have nokogiri check there for shared libraries first.

See: libxml2 for Nokogiri in JRuby

Nokogiri uses Ruby FFI to dynamically load native C code and FFI makes use of dlopen to do the actual loading of dynamic libraries. On OSX, dlopen searches for files specified by a couple of environment variables (LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH), and the current working directory. Setting LD_LIBRARY_PATH to /opt/local/lib worked for me. There may be differences in the environment variables used for dlopen on different platforms, so a look at the MAN pages would be a good idea if things don’t seem to work.

Running the rspec tests

JRuby invocation note: use this command prefix to run the rake spec tests from JRuby if you have a more recent version of libxml2 installed with macports:

LD_LIBRARY_PATH=/opt/local/lib jruby -S rake spec options

Running all the rspec tests:

  rake spec

Running a single file:

  rake spec SPEC=spec/routing/dataservice/bundle_contents_routing_spec.rb

Running a single directory:

  rake spec SPEC=spec/routing/dataservice

Running all the controller tests:

  rake spec SPEC=spec/controllers

Running the feature tests with cucumber

JRuby invocation note: use this command prefix to run the rake spec tests from JRuby if you have a more recent version of libxml2 installed with macports:

  LD_LIBRARY_PATH=/opt/local/lib jruby -S

Running all the feature tests:

  rake cucumber

Running all the feature tests using the ci_reporter gem that’s used on the hudson CI system:

  rake hudson:cucumber

Running a single feature:

  rake cucumber FEATURE=features/student_can_not_see_deactivated_offerings.feature

Understanding the Codebase

Themes

We are using theme_support plugin — except it doesn’t play very well with compass and asset_packager. So, we handle sass/stylesheets free-hand.
Here is a short description of how to setup a new theme:

  1. Copy an existing theme from RAILS_ROOT/themes/
  2. modify settings.yml to and add a theme: theme_name entry
  3. Put sass files in public/stylesheets/sass/themes//*.sass
  4. These sass files will be compiled and moved into public/stylesheets/themes//*.css
  5. Configure your themes assets in asset_packages.yml under the stylesheets section with the format:
 
  - <themename>:
    - themes/<themename>/style_name
      - (others)
eg: 
 
  - itsisu
    - themes/itsisu/itsisu
    - accordion

Some video walk-throughs

The Page Elements Model Part I:

screencast: The Page Elements Model Part I

Install github version of railroad with aasm patches from ddolar’s repo

Generate a graph of the projects models using railroad:


  railroad -o models.dot -M

Open that file (models.dot) with omnigraffle, or traslate to some other image format using the dot tool.

Page Elements Model Part II:

screencast: Page Elements Model Part II

Using mysql query browser (part of the osx mysql binary distrobution) to view schema:
Mysql gui-tools

Use the generator to generate page elements eg:


  ./script/generate element xhtml content:text

PageElement View partials:

screencast: PageElement View partials

Shows the relationship between:

  • pages/show.html.haml
  • pages/element_container.html.haml
  • shared/_embeddable_container.html.haml
  • /_show.html.haml

HAML, Compass and SASS:

screencast: HAML, Compass and SASS

Brief introduction to the technologies generally, and how we use them specifically

Javascript use in Portal

screencast: Javascript use in Portal

Javascript librararies we are using, and what things we have written by ourselves;
Stuff we did:

Capistrano Recipies (and problems)

warning: sometimes the permissions on the shared/cache/.git directory get changed to be read-only for the group. So what i do to fix this is:


  sudo chmod -R g+w ./cached-copy/

More info on capistrano-ext/multistage deployments can be found here: http://weblog.jamisbuck.org/2007/7/23/capistrano-multistage

Database recipes

There are a few database oriented recipes which should cover the basic db-oriented tasks you might want to do.
Here are some scenarios and how to do them. Remember, all commands are entered on your local machine, in the application root directory.

There are 4 basic commands:

  • rake db:dump
    • This dumps the current environment’s database to db/… eg db/production_data.sql or db/development_data.sql
  • rake db:load
    • This overwrites your current db with a sql dump from db/ into the current environment’s db, provided a dump exists for your current environment
  • cap (production|staging|development) db:fetch_remote_db
    • The same as db:dump, except it dumps the database from whichever remote instance you chose
  • cap (staging|development) db:push_remote_db
    • Same as db:load, except the remote database is overwritten

Download the production database to use locally


  cap production db:fetch_remote_db
  cp db/production_data.sql db/development_data.sql
  rake db:load

Reset the Staging or Development database with Production’s version

  • cap production db:fetch_remote_db
  • cap staging db:push_remote_db or cap development db:push_remote_db

Update Staging or Development with a copy of your local database

  • rake db:dump
  • cap staging db:push_remote_db or cap development db:push_remote_db

Note: For safety, you can’t push a database to production. If you accidentally try, you’ll get a message saying you can’t do that.

Deployment recipes

There are three custom deployment-oriented cap tasks defined in config/deploy.rb One works, One sometimes works, and the first one doesn’t!

  • cap deploy:from_scratch does not work.
    • It would need to be able to let users answer questions.
    • Instead of dropping the database, we would have to simply drop tables. (rails user does not have perms to drop dbs)
  • cap deploy:install_gems does seem to work, unless there are dependancies on the gem being installed.
  • cap deploy:shared_symlinks does work, and it links in the shared resources that we use. (like config/database.yml)

Workarounds for cap deploy:from_scratch: I usually just end up logging in to otto, dropping all the tables,
and then running the various rake tasks by hand:


  rake db:migrate
  rake rigse:setup:default_users_roles
  rake rigse:setup:create_additional_users
  rake rigse:setup:import_gses_from_file

We should solve this one sooner rather than later.

Miscellaneous recipes

Set the gse_key field for existing GradeSpanExpectations

  • cap convert:set_gse_keys

Updating a staging server.

Once a development branch has been deployed to a development server, tested
and found reliable enough to deploy to staging here’s how to do it.

Our convention is to create dev, staging, and production branches in the git repository following that use the same name.

For example the xproject family have the following capistrano stages and branches in teh git repository:

  • _xproject_dev__
  • _xproject_staging__
  • _xproject_production__

In the code below I will assume that we are using the xproject series of stages and branches.

If you don’t already have a local branch of staging


  git branch --track xproject_staging origin/xproject_staging

Switch to the staging branch and merge from xproject_dev

    
  git co xproject_staging
  git merge xproject_dev

Push your copy of the staging branch to the gihub repository:

  git push origin xproject_staging

Dump the production database to this file db/production_data.sql on the production server,
download it to the local folder db/production_data.sql, the cleans up the production db/ folder.

  cap xproject_production db:fetch_remote_db

Push the production database from the local db/production_data.sql to the staging server, then import
the data into the database on staging, then cleanup.

  cap xproject_staging db:push_remote_db

Run any migrations on the staging server:

  cap xproject_staging deploy:migrate

There may be rake tasks that need to be run to update or fix data in the database.

These should have corresponding capistrano tasks.

Here’s an older example:

If the ITSI importers have been updated and you want to run them again:

    
  cap xproject_staging import:erase_and_import_itsi_activities
  cap xproject_staging import:erase_and_import_ccp_itsi_units

Test the staging server: http://xproject.staging.concord.org/

If the authors confirm that there are no blockers then let people know when the
update will take place and perform these tasks on the production server.

other Rake tasks:

  1. rigse:make:investigations
    This task simply finds all activities with no parent investigation, and creates a new investigation for that activity. The created investigation has the same name and description as the activity it contains.

haml

We use haml for some templates, see: http://haml.hamptoncatlin.com/

To install this plugin we followed this procedure:

  1. gem install --no-ri haml
  2. haml --rails path/to/rigse_app

Creating the sakai integration demo user accounts.

OCT 7 2009 This documentation should go away as soon as we no longer need to import sakai users by hand.


  RAILS_ENV=production script/console
  require 'lib/sakaidemo'

Importing Sakai usrers from RINET / RIEPS.


  RAILS_ENV=production script/runner "(RinetData.new).run_scheduled_job"

Rendering

haml

We use haml for some templates, see: http://haml.hamptoncatlin.com/

to install this plugin we followed this procedure:

  1. gem install --no-ri haml
  2. haml --rails path/to/rigse_app

Rendering OTML

The sass template rites.otml.sass generates the css file: rites.otml.css which is used for styling the xhtml content in
OTCompoundDoc elements.

The Java OTrunk system uses Java html editor kit for rendering xhtml and implements a very limited version of CSS that is somewhere
between CSS1 and CSS2. You can find out more about this implementations limitations here: Java 1.5: Class CSS

Installers

Building installers requires that you are running on a mac with a local installation of Bit Rock
The rake tasks assume that bitrock is in the standard /Applications/ folder. You can override this by setting an
environment variable in your shell which points to the correct path, eg: export BITROCK_INSTALLER=/path/to/bitrock.app

config/installer.yml

Every host should have its own config/installer.yml file. There is a config/installer.sample.yml file which can help get you started.
The shortname field should be specific to that host. Because of limitations in bitrock, the shortname can not use spaces,dashes,underscore, &etc.
The jnlp_url should point to a jnlp url on the target host. When you run the rake build:installer:build_all or
cap installer:create tasks, the jnlp_url must be available.

Installer Rake Tasks:

Most of the installer building happens via rake tasks defined in lib/tasks/make_installers.rake. a complete list of tasks can
be gotten using: rake -T installer

Here are the two useful tasks:
<pre> rake build:installer:build_all # build all installers rake build:installer:new_release # create a new release specification interactively </pre>
Assuming that your installer.yml file is correct, running rake build:installer:build_all will take care of the rest.
Build_all will automatically clean up, recache jars, and bump version numbers.

Installer Capistrano Recipes

There are two cap recipes in config/deploy.rb which take care of creating installers using remote hosts installer.yml files.

  • cap installer:copy_config copies the local installer.yml to the remote server. This would be useful if you ran new_release locally, and then
    wanted to copy those config settings to the remote server.
  • cap installer:create creates the installers and updates the remote installer.yml file, and deploys the installer images.

Sample session for building installers:

boot strapping an unconfigured server:

In this session we are assuming that we are working with a host which does not have a local installer.yml file.
First we create a new local release. The first rake tasks asks a bunch of questions, which are answered from the point of view of the staging server.

<pre> rake build:installer:new_release cap staging installer:copy_config cap staging installer:create </pre>

After running rake build:installer:new_release we end up with the following local installer file:
<pre> shortname: RitesStaging version: "200912.00" jnlp_config: http://rites-investigations.staging.concord.org/investigations/545.jnlp <code><pre>

this file gets pushed up to staging. with cap staging installer:copy_config we only have to do that the first time we create an
installer on staging. We could just as easily edit config/installer.yml.

The cap staging installer:create handles incrementing the version number, and pushing the new config files and installers onto staging.

creating a fresh installer for a host that has had installers before:

<pre> cap staging installer:create </pre>

not much to do.

Authentication, Sessions, and Cookies

RESTful Authentication

RESTful Authentication is already setup. The routes are setup, along with the mailers and observers.
Forgotten password comes setup, so you don’t have to mess around setting it up with every project.

The AASM plugin comes pre-installed. RESTful Authentication is also setup to use user activation.

Uses the Database for Sessions

Bort is setup to use the database to store sessions by default.

User Roles

Bort, as of 0.3, has Open ID integrated with RESTful Authentication. Rejoice!

However I’ve (stepheneb) removed this to make debugging the authentication simpler and it
seems as though open-id is not becoming widespread.

Will Paginate

We use will_paginate in pretty much every project we use.

Exception Notifier

You don’t want your applications to crash and burn so Exception Notifier is already installed to let
you know when everything goes to shit.

config/initializers/exception_notifier.rb does the setup. Currently it reads
“admin_email” from config/settings.yml and use it as the destination address.
The setup can be modified to include multiple email addresses.
See the homepage readme of exception notifier.

Bug

It seems rails 2.3.3 and 2.3.4 fails to deliver emails when someone passes
multiple destination addresses as an array, which exception notifier does.
config/initializers/fix_mailer_on_rails_2.3.4.rb fixes the problem.

The code is borrowed from Dmitry Polushkin

Databases

If you are using mysql 5.5 or later you’ll need the mysql2 gem. This will be enabled by bundlers Gemfile if you define the environment variable:
RB_MYSQL2=true

On OS X the mysql2 gem usually can’t find the mysql client library that it needs to run. The command below fixes that. It assumes your mysql is installed in the default basedir of /usr/local/mysql/lib. And it assumes you are using bundler.
<pre> install_name_tool -change libmysqlclient.16.dylib /usr/local/mysql/lib/libmysqlclient.16.dylib `bundle show mysql2`/lib/mysql2/mysql2.bundle

CSS

Asset Packager

Portal authoring and portal site uses: Asset Packager docs

cap deploy will trigger the asset_packager to run.
you can also run locally by hand: rake asset:packager:build_all

Assets are configured in config/asset_packages.yml

Packages up your css/javascript so you’re not sending 143 files down to the user at the same time. Reduces
load times and saves you bandwidth.

Routing

Application Settings

Settings YAML

There is a settings.yml file that contains site-wide stuff. The site name, url and admin email are all used
in the RESTful Auth mailers, so you don’t need to worry about editing them.

Database YAML

Technical debt.

Here is a brief list of things which need to be looked into:

  • the embeddables should be dryed up with some mixin / super class.
  • not all embeddables are using send_update_events, which is causing stale pages.
  • ocassionally browser rendering gets wonky and raw html and or javascript get displayed in the page.
  • transition to unobtrusive JS.
  • send_update_events might not do what we want it to do, tests should be written for it.

Misc

  • password and password_confirmation are set up to be filtered
  • there is a default application layout file
  • a page title helper has been added
  • index.html is already deleted
  • rails.png is already deleted
  • a few changes have been made to the default views
  • a default css file with blank selectors for common rails elements

Credits

Bort put together by people at Fudge