up to: DevTools; see also: HeronQA

Here we elaborate on best practices enumerated in our CodeReviewNotes checklist. See DevTools#prog for intro/reference materials.

Writing quality code includes documentation and testing

Let's adopt Dan's 2006 new year's resolution:

no more undocumented, untested code. I like to write little ditties for stress relief, but without docs and tests, the code seems to add to my stress, over time. At my first job out of college, at Convex, management devoted equal resources to coding, documentation, and testing/QA. The more time goes by, the more wisdom I see in that way of working. If the code is worth writing, it's worth testing and documenting.

It's best to write the tests first, and it's good to have use cases in mind, if not documenation written down, before that.

For personal projects, I'm not disciplined enough to write the tests first every time; if the code I write works the first time, I sometimes let myself get away with it. But I'm doing pretty well about doing test-driven debugging, at least. I write tests for any code that doesn't work the first time. And I write tests when I refactor and change things. The confidence to make changes that comes from having tests in place is very freeing.

Python best practices: doctest, and PEP8

The python community has pretty strong norms; for example, the style guide, PEP8, is supported by various tools and editors.

Python's doctest rocks, because it makes developing tests and documentation while you code second nature.

more on nose, Jenkins...

Don't mock your own code

TODO: elaborate

Borrowing from the Python Package Index

We rely on carefully selected packages from the the Python Package Index (aka pypi or the Cheese Shop) such as cx_Oracle, injector, SQLAlchemy, and pyramid. (Yossi Kreinin wrote a nice piece on build-vs-buy in 2008).

todo: organized platform-specific details for setting up easy_install/pip between this page, PythonOnWindows#pypi and PythonVirtualEnvironment#pypi

easy_install seems to be fading out of fashion in favor of pip, though I haven't learned about any significant differences, yet.

We're exploring various deployment techniques (#1242), but one thing is pretty clear: for any packages that are not pure python, build an .rpm package and install that. We shouldn't need a C compiler on production machines.

For installing python libraries on a box that does not have http/s out rights: (wheel / no-index / download)

pip SSL woes


cat /etc/ssl/certs/DigiCert_* >cacerts.crt
pip --cert cacerts.crt install supervisor

context: ticket:3951#comment:8

SQL stuff

We're still sort of fumbling through this. See also:

Java and Scala

For what it's worth, we have some notes on JavaDevTools.

For fun, see also:

R community norms for reusable code (packages)

The equivalent to pypi for R is CRAN. The official documentation for developing packages is Writing R Extensions. It's a little klunky. Hadley's book, Advanced R programming, is more concise, but it's in-progress and unreliable in some details.

I (Dan) am trying to use automated tests and our continuous integration service to manage risks of refactoring source:rgate for scalability (RGateDataFlows). The idea is to use the testing hooks in R CMD check.

The development workflow based on Hadley's devtools package is nice, but it clashes a little bit with Developing Packages with RStudio, especially the "knit" button.

todo: CITATION for cdrbroker package

knitr, an R work-alike for python's doctest

Hadley's book, Advanced R programming, recommends roxygen for documentation and testthat for unit testing, but every time I write a test description, the redundancy w.r.t. documentation grates on me. There is support for using examples in R documentation as automated tests, but I've become somewhat addicted to RStudio's support for knitr.

In R 3.x, you just stick the .Rmd files in vignettes/ and R CMD build takes care of the rest (pretty much). But we're stuck with R 2.15.x (#2418), which requires kludges involving a Makefile and a fake .Rnw file, and I haven't found a way to arrange the files to work in both environments.

add notes on configuring knitr to fail-stop.

TODO: Get Jenkins to build and publish documentation from roxygen comments and knitr. See Build Static HTML Help Pages for R Packages.

Road not taken: Namespace Management a la Javascript

Dan tried this out but didn't continue due to lack of support from R dev tools (R CMD check) and distance from community norms.

in mod_x.R:

mod_x <- function() {
  f1 <- function(arg1) { ... }
  f2 <- function(arg1, arg2) { ... }
  # this is where we list our "exports"
  list(f1=f1, f2=f2)

then to use it:


Raw Notes

Migration from source:rgate/rgate/cdrbroker/r_dev_notes.Rmd is in progress; see also R bookmark tag (and programming, quality, statistics)


but i got:

Updating cdrbroker documentation
Loading cdrbroker
Error in parse(text = lines, n = -1, srcfile = srcfile, keep.source = TRUE) : 
  unused argument(s) (keep.source = TRUE)

link from README to rgate...

... lots of stuff elided... see #R logs and my diigo library...

<DanC_> AARG! my S4 method works once, but later in the same session I get: unable to find an inherited method for function "dbGetQuery", for signature "MockConn", "character"
 ah. one shouldn't source() a file from the package one is testing.
<bosie> DanC_: hence one uses devtools
<bosie> ;)
<DanC_> yes, I'm using devtools. But it doesn't prevent me from bonehead plays such as source()
 and the resulting diagnostics are, to say the least, obscure
 ## huh?
 Error in FUN(X[[1L]], ...) : 
  promise already under evaluation: recursive default argument reference or earlier problems?
  Arguments of the form x=x cause this.

PHP: Abandon hope all ye who enter here

There are some best practices for documentation and testing in the PHP community, but the portion of the community that follows them is so small that they could hardly be considered norms. PHP: a fractal of bad design from April 2012 gives a sense of the challenges.

But we can't avoid PHP (e.g. UsingDrupal and REDCap). We try to do black-box testing with selenium to manage the risks.

Last modified 2 years ago Last modified on Feb 1, 2019 10:02:14 AM