Weblocks User Guide

This is the user's guide to Weblocks.

http://teddyb.org/rlp/tiki-index.php?page!Learning+About+Weblocks

1 Introduction

1.1 Getting Support

There is both excellent community and professional support available for Weblocks.

Community support is available via the Google group at http://groups.google.com/group/weblocks.

For commercial support and consulting contact Leslie P. Polzer <polzer@gnu.org>.

1.2 Installation

1.2.1 Setup your Common Lisp environment

You can skip this section if you already have a comfortable Common Lisp environment.

1.2.1.1 Common Lisp implementation

First you must choose a Common Lisp implementation.

Weblocks is designed and implemented to be portable and should run on the most popular Common Lisp implementations. It is currently tested best on SBCL and Clozure CL, though.

1.2.1.2 Development environment setup

There are at least two fundamentally different development approaches for using Common Lisp:

1. Editor-centric development: you access all Lisp functions from within your editor.

   Example: Emacs/Slime, Vim/Limp, Lispworks/IDE.

   Incremental development happens mainly on the S-Expression level. This means that you edit a SEXP and send it with the help of editor directly to your Lisp image, which evaluates it, thus affecting the current Lisp environment.

2. UNIX-style development: one tool for each job. The editor is not all that important here (as long as you're comfortable with it and it supports at least a basic level of paren highlighting.

   Example: Vim and your favorite terminal emulator. You start Vim in one window and your Lisp in another. Interaction happens by reloading your applications ASDF system and simple copy/paste of snippets.

We will try to be largely agnostic of the development approach in this manual which actually means that we tend towards the second approach: Lisp calls are referred to by what you'd type in your REPL, not by Emacs shortcut as it is often the case.

For a basic comfortable SBCL setup, see Appendix A.

1.2.1.3 Installation

There is a comprehensive installation guide on the main web site:

http://weblocks.viridian-project.de/installation

1.3 Setting up a project

Weblocks comes with a simple application template generator.

Start your Lisp and run

(wop:make-app 'NAME "DIR")

Another effective way to hack on a new application is copying the code of an application from the examples/ directory.

1.4 Components of Weblocks

1.4.1 Widgets

Widgets are the main building blocks of Weblocks (hence the name).

At the heart of Weblocks is a tree¹ of widgets that is manipulated by the clients requests. When a client sends its first request to a Weblocks application then a new session is started for it and a widget tree is associated with this session.

• ¹An acyclic graph with exactly one parent per node

This initial widget tree is computed as defined by the application developer. A special function (usually called init-user-session) is called by Weblocks to initialize a new session. This function then proceeds to set up a widget tree by adding children to the root of the widget tree (accessible with the macro root-widget).

The initial content and structure of the widget tree may depend on arbitrary factors; it is conceivable (although probably not very sensible) to generate different widget trees depending on the time of day.

A client's request for a specific URI modifies the widget tree: widgets called dispatchers choose their one child based on the current request URI.

There is a convenience function make-widget that creates widgets from strings (anything that can be printed, in fact) and functions.

1.4.2 Actions

Apart from session initialization the widget tree may be modified by actions.

Actions are plain functions that are stored in a session hash table the keys of which are unique identifiers. This way they can be called by the browser.

When Javascript is available actions will be called via AJAX, otherwise a normal request will be initiated.

It is often useful to set up closures as actions.

Example of typical action usage:

(defwidget counter ()
  ((count :accessor count-of :initform 0)))

(defmethod render-widget-body ((widget counter) &rest args)
  (with-html
    (:p (esc (format nil "The counter is at ~D." (count-of widget))))
    (:p (render-link
         (lambda (&rest args)
          ;; closes over WIDGET.
          (incf (count-of widget)))
         "Count up!"))))

1.4.3 Navigations and dispatchers

Dispatchers are widget that configure themselves and their children (i.e. broadly speaking the widget tree) based on the URI of the request.

Navigations are dispatchers that maintain a simple one-to-one association between an URI token² and a widget.

• ²each path component of an URI is a token; for example in /foo/bar/quux there are three tokens foo

Note that each widget can also in turn be a navigation that consumes an URI token, thereby building a many-to-many relationship between tokens and widgets.

The function make-navigation is a convenience frontend for building navigations.

The first argument is a title for the navigation. The rest of the arguments map tokens to widgets:

(defun init-user-session (root)
  (setf root (make-navigation "Food Shop"
                              "Fruits" (make-fruit-navigation)
                              "Vegetables" (make-vegetables-navigation)
                              "Billing" (make-instance 'billing-widget :shop-type 'food))))

• ³i.e. symbols naming a global function and function objects themselves

1.4.4 Views

Views provide convenient ways to define how data object are to be rendered.

Weblocks currently comes with three types of views: data views, table views and form views.

Form views are especially useful because they let you build forms in a declarative manner, error checking and reporting included.

Views mainly consist of a rendering framework unique to the view⁴ and a series of view fields.

• ⁴The table view for example knows how to render a HTML table.

View fields usually map to the slots of your data class but they are flexible enough to be used in any other way. Associated with each view field regardless of the view type are things like a human-readable label and information on how to render the field (presentation).

Form views include additional parameters like how to translate user input to a proper model value (parser) and constraints on the data (satisfies).

1.5 Stores

Weblocks offers a storage framework with several backends. At the moment the supported backends are Elephant, cl-prevalence, CLSQL and in-memory storage.

You are free to use your own storage mechanisms, however.

Appendix C offers a comparison of the different stores.

1.5.1 Applications

One Lisp image may serve multiple web applications at once.

The target web application of a request can be determined based on the hostname, the port or the URI for example.

You don't have to worry much about this if you only intend to run a single application.

1.5.2 HTML and Javascript generation facilities and utilies

Weblocks comes with CL-WHO and Parenscript, two powerful packages that provide Lisp compilers for HTML and Javascript.

This means no more munging around with strings, which in turn means

• no more annoying mistakes like forgetting a semicolon or forgetting to close a tag
• no need to remember infix precedence rules
• make full use of Lisps selective quoting facilities

Javascript may still be written as simple strings however. This comes in handy sometimes when working with a larger example copied from a Javascript library manual.

And you're free to use other facilities as you prefer. For example YACLML⁵ is a function-based HTML generation facility that you might want to take a look at.

• ⁵Maciej Pasternacki wrote an excellent introductory article about YACLML where he also compares it with CL-WHO

1.5.3 Continuation-based tools

Some neat features of Weblocks are built on continuations. But don't worry, you will only need some high-level understanding of continuations to use these features along with some rules of thumb.

2 Widgets

2.1 Building a widget

Most widgets consist of two main parts:

1. a class definition using the defwidget macro.
2. a method for render-widget-body specialized on the widget's class.

2.2 Rendering protocol

Map over the currently active (by URI) widget tree:

  render-widget -> render-widget-body (widget-update-children?)

3 Forms

Forms enable the user to communicate with a web application.

Usually the server side action boils down to selecting, modifying, creating or deleting data sets (this is sometimes abbreviated as CRUD: Create/Read/Update/Delete)

Building web forms is usually a cumbersome process. Elaborate but complicated solutions have been devised⁶, but so far we haven't found any of them to match the ease of use and flexibility of Weblocks' declarative view DSL.

• ⁶for example PHP's Quickform extensions

3.1 Introduction

The form mechanism consists of two parts, the dataform widget and the form-view view type.

Forms are usually built by defining form views using the defview macro and instantiating a dataform object with this view.

3.2 Simple example

Let's define a view for creating and editing bug reports.

Let the model be defined thus:

(defclass bug-report ()
  ((id :type integer :initform (create-unique-id))
   (status :type (member :new :open :resolved) :initform :new)
   (summary :type string)
   (body :type string)))

(defview bug-report-form-view
    (:type form :inherit-from '(:scaffold bug-report)
     :caption "Enter a bug report")
  (id :hidep t)
  (status :hidep t))

Let's use this view to derive the view for developers:

(defview bug-report-form-view/developer
    (:type form :inherit-from 'bug-report-form-view )
  (status :hidep nil))

You can define scaffolding rules for your own types.

As part of the validation process Weblocks will also check whether a user input matches the slot's type regardless of whether you use scaffolding or not.

But let's assume that we want custom labels for the dropdown:

(defview bug-report-form-view/developer
    (:type form :inherit-from 'bug-report-form-view )
  (status :hidep nil
          :present-as (dropdown :choices '(("This report is totally new, don't trust it!" . :new)
                                           ("Yeah okay, we're working on it." . :open)
                                           ("We've solved that problem already..." . :resolved)))))

3.3 Quickforms

quickforms are specialized dataforms.

They provide a way to build forms based entirely on a view; they are handy for operations where you don't have the user working on an actual model instance.

Let's dive right into it:

4 Views

4.1 Syntax of declarative view declaration

The macro defview is a declarative DSL for defining views.

Syntax:

 (defview (NAME [:type TYPE] [:inherit-from INHERIT-FROM]
                [:satisfies SATISFIES] VIEW-KWARGS...)
   [FIELD-NAME
    | (FIELD-NAME [:type FIELD-TYPE] [:initform INITFORM]
		  [:present-as PRESENT-AS] [:parse-as PARSE-AS]
		  FIELD-KWARGS...)]...)

In the above form, these metasyntactic variables have the following values:

NAME

When non-nil, an unevaluated unique symbol identifying the view to other parts of the system that require a view, resolving it using find-view.

TYPE

An unevaluated designator for the class of the resulting view (determined by view-class-name), and half the designator for the class of this view's fields (determined by view-field-class-name applied to view-default-field-type). Defaults to data.

INHERIT-FROM

A designator for view slot inherit-from. Forms like (:scaffold DATA-CLASS) will replace :scaffold with TYPE. See find-view for further transformation.

Evaluated.

SATISFIES

A designator for form-view slot satisfies.

Evaluated.

VIEW-KWARGS

Other arguments passed directly to make-instance for the resulting view, quoting subject to view-argument-quoting-strategy.

Argument semantics:

FIELD-NAME

An unevaluated designator for view-field slot slot-name.

FIELD-TYPE

A designator for the class of the field, in combination with TYPE, as explained above for TYPE.

FIELD-KWARGS

Other arguments passed directly to make-instance for the resulting view field, quoting subject to view-argument-quoting-strategy.

However, field-kwargs keywords present in *custom-view-field-argument-compilers*' are not included in the call to make-instance; see that variable for information on how those are transformed.

The built-in custom view fields are as follows:

INITFORM

A form for mixin-view-field slot initform (which is really an initfunction).

PRESENT-AS

A designator for view-field slot presentation. The symbol or CAR is mapped to a class name through presentation-class-name, and the CDR, a series of keyword arguments, is passed to make-instance on said class, subject to view-argument-quoting-strategy.

PARSE-AS

A designator for form-view-field slot parser. Otherwise similar to PRESENT-AS, but mapping class names using parser-class-name."

4.2 Configuring view fields at run-time

4.3 Customizing scaffolding

4.4 Limitations

4.4.1 Available presentations

4.4.2 Available parsers

5 Continuation-based tools

5.1 A high-level explanation of continuations

5.2 How Weblocks uses continuations

Only to replace widgets and remember how to restore the original state. XXX VERIFY

5.3 Continuation-based tools

5.3.1 Widget replacements

5.3.2 Modal dialogs

5.4 When to use with-call/cc and lambda/cc

6 Dependency handling

7 HTML generation

At the moment only CL-WHO is supported out of the box, but YAML and others may also be used.

7.1 CL-WHO

examples to point out pitfalls

7.2 Utilities

7.2.1 WITH-HTML-FORM

8 Working with continuations

8.1 Introduction

8.2 Widget continuations

8.3 Using the high-level interfaces

9 Adapting Weblocks to your needs

9.1 Introduction, Hooks

Every GF is a hook: subclass, :before, :around, :after

9.2 Writing a presentation

9.3 Writing a form field parser

9.4 Writing a dispatcher

9.5 Integrating 3rd party Javascript

9.6 Deploying multiple Webapps

10 Weblocks Internals

Not only useful for those wishing to hack the internals but also to gain an understanding of Weblocks that aids in debugging hard problems.

10.1 Request handler

10.2 Dispatching mechanism

10.2.1 Widget tree update mechanism

The whole tree update protocol goes like this:

1) HANDLE-NORMAL-REQUEST calls UPDATE-WIDGET-TREE, which walks the tree using WALK-WIDGET-TREE starting at ROOT-WIDGET and calling update-children at every node.

2) The selector's UPDATE-CHILDREN method (valid for all selectors, i.e. widgets that process URI tokens) calls GET-WIDGET-FOR-TOKENS.

3) if a widget corresponding to particular URI tokens is found, UPDATE-CHILDREN calls UPDATE-DEPENDENTS, so that the selector (or its subclass) may update its dependents list and do other housekeeping. The default implementation of UPDATE-DEPENDENTS just calls (SETF WIDGET-CHILDREN) to store the children under the :SELECTOR type.

Usually the only thing you'll want to do if you are implementing your own kind of selector is to subclass selector and provide a GET-WIDGET-FOR-TOKENS method for it. See class ON-DEMAND-SELECTOR for an example."))

10.2.2 Selectors

(defgeneric get-widget-for-tokens (selector uri-tokens)

(:documentation "Given a list of URI tokens, map them to a widget. All selectors implement this method. There can be multiple strategies for mapping URI tokens to widgets: static maps, dynamically-generated widgets, dynamically-generated widgets with caching. Returns a widget or NIL if not found. Modifies URI-TOKENS.

10.3 Views

views are the second highest level mechanism (after forms) in the weblocks system of forms, views, presentations & parsers that form the rendering framework for structured data. views are defined using defview.

defview is a macro that in turn calls defview-anon. defview-anon is not visible outside weblocks — if you have to use it, call it with weblocks::defview-anon. defview-anon actually makes the view, defview attaches a gensym'd name to it and stores it in the global views hash.

so all views are global, and any changes made to one show up in all the others (at least after a refresh of the browser).

10.4 Form pipeline

11 Contributing to Weblocks

1. checkout branch
2. make changes

11.1 test changes (how?)

1. add tests (where?)
2. make patch
3. discussion/review

See tutorial at http://trac.common-lisp.net/cl-weblocks/wiki/WeblocksDevelopment

12 Hints

12.1 Basic SBCL setup

Appendix A: Basic SBCL setup

I recommend the following SBCL initialization file (on UNIX systems it is ~/.sbclrc):

; stale FASLs
(defmethod asdf:perform :around ((o asdf:load-op) (c asdf:cl-source-file))
    (handler-case (call-next-method o c)
      (#+sbcl sb-ext:invalid-fasl
       #-(or sbcl allegro lispworks cmu) error ()
       (asdf:perform (make-instance 'asdf:compile-op) c)
       (call-next-method))))using aclrepl (see also (http://www.sbcl.org/manual/sb_002daclrepl.html))

12.2 Debugging Techniques

TRACE (beware builtin funs), BACKTRACE, DESCRIBE If no errors show up (sometimes occurs e.g. when working with continuations!) try CATCH-ERRORS-P

12.3 Deployment

detachtty screen tmux remote swank

12.4 Store comparison

Appendix C: Store comparison

Different stores have different merits. Here's a short comparison.

12.4.1 cl-prevalence

This library embodies the prevalence database strategy: Everything is kept in main memory; all writes are logged to disk, and the in-memory database is restored with the aid of the log at initialization time.

12.4.1.1 Advantages

• This means excellent speed and seamless integration with CL as query language.

12.4.1.2 Disadvantages

• No transaction support for objects in memory.
• Database size limited by size of virtual memory.
• No load balancing capabilities (but you can run the database on a separate machine)

12.4.1.3 Bottom line

Prevalence is a great library to start with because you can do carefree prototyping and also run production sites.

Important note: The original version of cl-prevalence has some bugs that prevent is from working reliably when used with Weblocks.

It is recommended to use the fork at http://bitbucket.org/skypher/cl-prevalence/ where this problem has been remedied.

12.4.2 Elephant

Elephant is based on a key-value database model (like SQL servers) It offers convenient CLOS integration and decent speed.

12.4.2.1 Advantages

• Transactions affect in-memory objects.
• Excellent support for schema updates: Elephant manages most issues on its own even across process boundaries. Special cases (slot migration/deletion) can be specified by the user by hooking into the MOP.
• Queries are formulated completely in CL.

12.4.2.2 Negatives

• Less speed than cl-prevalence especially with a high number of slot accesses.
• Watch out when handling primitive types like arrays and conses, Elephant doesn't have its hooks in there.
• BDB backend: load balancing not easily realized.
• PostgreSQL backend: incomplete support for sorting values containing NIL.

12.4.2.3 Bottom line

Not as easy to set up as cl-prevalence, but a great library to work with.

12.4.3 CLSQL

The CLSQL store relies on the popular SQL ecosystem. This means proven client-server management systems like PostgreSQL, hosts of excellent tools and great interoperability.

On the negative side SQL has an impedance mismatch when matched against object-oriented data structures.

You cannot use CL as a query language either, CLSQL only provides a Lispy layer over SQL.

12.5 Weblocks compared to other frameworks

12.5.1 At a glance

12.5.2 In-depth comparison

Weblocks

Django: read basics at djangobook.com

Django works at a level similar to hunchentoot. A series of urls is mapped to functions. Templates can be used to fill in html.

Uses mvc.

Python: Twisted?

Google App Engine?

Rails: Uses mvc. Ruby has some nice language features. Framework is backwards, and forces you to repeat yourself many times. A guiding

principle is to not repeat yourself (colloquially DRY) and in combination with a poor DSL it can be quite gibberishy. Strongly tied to SQL as data store.

Ruby: Other?

Perl: Mason?

PHP: Cake?

Seaside