----------------------- REVIEW 1 ---------------------
PAPER: 26
TITLE: LiquidHaskell: Refinement Types in the Real World
AUTHORS: Niki Vazou, Eric L. Seidel and Ranjit Jhala

OVERALL EVALUATION: 2 (strong accept)
REVIEWER'S CONFIDENCE: 4 (high)

----------- REVIEW -----------
Liquid types provide a lightweight mechanism for expressing contracts and other logical properties, including termination and totality, within Haskell. These types can be verified automatically, by means of SMT solving.

The paper can be seen as a companion to the ICFP'14 paper "Refinement types for Haskell", which covers the theory of the approach. This paper covers a number of examples, as well as giving a general introduction to the approach. It also covers (in common with the ICFP paper) a set of benchmark results and related work. The paper is well written, and gives a set of well chosen examples.

One piece of terminology is puzzling: measure is introduced as something that gives the size of a value, but it is later used for more general properties e.g. almostRB over red-black trees.

The emphasis on integer values is also apparent in the predicate for non-emptiness for lists, which is expressed by saying that its length is greater than zero. Some rationale for this would be useful.

Sections 3 and 4 cover "totality" and "termination" respectively. It would be helpful to explain these terms at the start of section 3, so that the two are distinguished appropriately.


----------------------- REVIEW 2 ---------------------
PAPER: 26
TITLE: LiquidHaskell: Refinement Types in the Real World
AUTHORS: Niki Vazou, Eric L. Seidel and Ranjit Jhala

OVERALL EVALUATION: -1 (weak reject)
REVIEWER'S CONFIDENCE: 3 (medium)

----------- REVIEW -----------
Summary:

This paper presents LiquidHaskell, a implementation of Haskell
extended with refinement types.  In particular, LiquidHaskell is a
refinement type checker that can be run on a Haskell source program
augmented with refinement type annotations and will report whether the
program meets the specification given by the refinement types.  The
paper gives a tour of features of the language, with a detailed
discussion of the kinds of properties that it is possible to verify
with refinement types, what kinds of annotations are necessary, and
how one can go about verifying rich properties of Haskell programs,
such as memory safety, with the LiquidHaskell system.

Pros:

I found this paper to be an engaging and pleasant read.  The
example-driven approach does a great job of illustrating what's
possible with LiquidHaskell (and quite a lot is possible).  The
examples were clear and easy to follow, and from reading the paper I
gained a better understanding both of the LiquidHaskell system and of
refinement types in general.

Cons:

The main flaw I see with the paper as it stands is that it does not
make clear what the relationship of this work is to the authors'
related paper appearing in ICFP '14, and what the novelty of this
paper is in comparison to it.  For instance, the evaluation section of
this paper duplicates some of the evaluation section in the ICFP '14
paper.

Although I enjoyed reading the paper, I'm not sure that it is enough
of a research contribution on its own.  However, I think it could be
successfully recast as an experience report.

The paper is also missing any discussion of how the LiquidHaskell type
checker itself is implemented.

Minor presentational issues:

  * I wondered what "measures" were when they were mentioned at the
    start of section 2; consider at least adding a forward reference
    to section 2.3.

  * Although the abstract mentions "a combination of refinement types
    and SMT solvers", the paper never mentions SMT solvers again
    except for a brief mention of Z3 in passing on page 9 and in the
    related work on page 11.


----------------------- REVIEW 3 ---------------------
PAPER: 26
TITLE: LiquidHaskell: Refinement Types in the Real World
AUTHORS: Niki Vazou, Eric L. Seidel and Ranjit Jhala

OVERALL EVALUATION: 1 (weak accept)
REVIEWER'S CONFIDENCE: 4 (high)

----------- REVIEW -----------
This paper is a pleasing overview of the practical use of Liquid Haskell to verify large, real programs.

Overall, I think this is a wonderful contribution to the Symposium.  My only concern is that the paper appears to be mostly an experience report (is this true?).  If so, I would appreciate a little more high-level guidance and advice for non-experts to use LH.  For example, are there lessons learned in debugging the (not-so-friendly) SMT output?  Is there anything you couldn't verify (e.g., libraries that use non-linear arithmetic or alternating quantifiers in the specifications?  If not, why?

I think LH might give us a road map to have a verified set of base libraries for Haskell, which is quite an achievement no other mainstream language!

Minor comments:

- It appears this paper is an experience report, and I encourage the authors to mark it as such.

- "we see that the output type is essentially" Can you add a footnote why it's not this type exactly?

- "global in that, LIQUIDHASKELL" --> no comma

- You mention that totality checks are w.r.t. Core.  Are errors then also w.r.t. Core?

- "This case study illustrates an advantage of LIQUIDHASKELL over specialized provers (e.g., catch)" --> can you provide a reference for catch?

- Before discussing HsColour, I'd give a brief explanation of what it is.

- For memory safety verification, do you have some sort of model of the runtime that you're using?  More generally, can you describe in more detail your approach to reasoning about non-pure code?

- Regarding your XMonad experience, you might want to compare your approach to the work of Wouter Swierstra: "Xmonad in Coq (Experience Report)".


----------------------- REVIEW 4 ---------------------
PAPER: 26
TITLE: LiquidHaskell: Refinement Types in the Real World
AUTHORS: Niki Vazou, Eric L. Seidel and Ranjit Jhala

OVERALL EVALUATION: 2 (strong accept)
REVIEWER'S CONFIDENCE: 3 (medium)

----------- REVIEW -----------
This paper describe the basics of LiquidHaskell (LH) and the authors' experience using LH to prove properties of several, mostly popular, libraries in the Haskell ecosystem.  LH leverages programmer-supplied annotations and type inferences to produce terms for an SMT solver, Z3 in this case, which indicate the presence or absence of program properties.

Much to the authors credit, though I've only lightly followed LH and refinement types this paper explained the critical features in a digestible manner.  However, the limitations of the general approach and tool in particular were more nebulous and less explicit.  Example surprising or grey areas include the inability to verify aspects of HsColour and exactly what the termination checker might say about code consuming finite portions of infinite streams (ex: `take n (map f [1..])`).  Also, an explanation of the amount of effort needed to verify termination of the 33% of functions for which default measures did not suffice would be welcome.

Given that verification of functional properties is the most immediately applicable section it is good to see real libraries considered.  It comes as a bonus that XMonad has already been formalized and properties proven in Coq, but the paper failed to capitalize by making any sort of statements contrasting the properties proven or expertise necessary for each verification - the question wasn't even acknowledged nor the prior work cited.

Overall, the paper was strong in specific constructs and experience when the tool is applied to already-proven libraries by expert users.  The paper would be even more useful if it could communicate the boundary of what is achievable, further reveal developer benefits in terms of bug finding or productivity (vs mostly validating pre-existing correctness properties), and identify experiences that required intimate knowledge of LH internals (if any).

Ignoring practical engineering matters, the applicability of LH to a project or problem is determined both by what the tool can do and by its limitations.  These two aspects bisect problems, cleanly or not, into those for which LH is a good fit and others.  Unless shortcomings are stated more explicitly it will remain hard for readers to understand when LH is a good fit for a particular problem.

Nits:

Abstract: The total line count is less interesting than the complexity of the properties and the complexity of the code base.  This is hard to quantify, but the single largest code base is a better estimate and the ranges of LOC and annotation count is even more informative.

1. Introduction: The definition of div is more restrictive than Prelude.div, this was mildly distracting.  While the example annotated type is instructive I can't help but feel a better example exists.

The document is not intended to describe the structure of LH, however many develops with correctness concerns continue to pick and choose their underlying prover carefully.  Whether or not the SMT solver is interchangeable interests audiences likely to put LH into practice.

2. LiquidHaskell

*Input* Item 2:  "a set of directories containing imported modules"

Doesn't this present a scaling issue?  More complex code management and build systems resulting in a less useful tool?

2.2 Verification

"If we replace the <= with a <" ...

Specifically, "the second <=".  There are two.

7. Discussion
Call out more of the weaknesses of the system.

* When applying LiquidHaskell to a given library were any improvements to the tool necessary to complete the analysis, or was the tool sufficiently complete such that other users (not the maintainers) could have performed the same analysis?

* How practical is organizing and keeping source copies of all imported modules?

* When writing code, how can a developer know what constructs will present verification difficulties of the sort in HsColour?
