pyKook Users' Guide

release: 0.7.2

Preface

pyKook is a software build tool such as Make, Rake, Ant, SCons or Cook. It is implemented in Python and runs any platform Python support. Basic command (copy, move, rename, mkdir, ...) is also implemented in Python and allows you to execute platform-depended command.

pyKook liken build process to cooking. Input file is called 'ingredient', output is 'product', task is 'recipe', build file is 'cookbook'. pyKook generates products from ingredients according to recipes. You describe products, ingredients, and recipes in cookbook.

Features:

• Impremented in pure Rython and runs any platform which Python supports.
• Input file (called 'cookbook') is named 'Kookbook.py', which is equivarent to 'Makefile' of Make or 'build.xml' of Ant.
• Cookbook's format is pure Python. You can write any Python code in kookbook.
• Supports command-line scripting framework.
• Supports command execution on remote machines.

Caution! pyKook is currently under experimental. It means that the design and specification of pyKook may change without prior notice.

Table of Contents

• Preface
  • Table of Contents
• Cookbook
  • Recipes
  • Timestamp and content
  • Product and Ingredients
  • Specific recipe and generic recipe
  • Conditional Ingredients
  • File Recipe and Task Recipe
  • Default Product
  • Properties
  • Materials
  • Command-line Options for Recipe
  • Load Other Cookbooks
• Other features
  • Category
  • Define Recipes Dinamically
  • Meta Programming
  • clean, sweep, and all recipes
  • Command-line Scripting Framework
  • Short command
  • Short notation
  • Debug mode
  • Invoke Recipes Forcedly
  • Nested Array
  • Cookbook Concatenation
• Remote Task Recipe
  • Ssh Configuration
  • Remote Object
  • Available Commands
  • Sudo Command
  • Password Object
  • Switching Remote Hosts
  • Deployment Example
  • Restrictions
• Trouble Shooting
  • xxx: product not created (in file_xxx())
  • *.c: can't find any recipe to produce.
  • sh: line 1: ingred: command not found
• References
  • Filesystem Functions

Cookbook

This sectipn describes how to write cookbook.

Recipes

Cookbook should contain recipes which are defined by function and decorators.

• @recipe creates new recipe from function.
• @product() specifies filename produced by recipe. This takes only an argument.
• @ingreds() specifies filenames required to produce a product. This can take several arguments.
• Function file_xxx() is called to product a product. This function is called as recipe method. Recipe method name should be start with 'file_' prefix if it produces file product.
• Function description is regarded as recipe description.

In cookbook, some helper functions provided by pyKook are available. For exaple, function 'system()' invokes OS-depend command. See References for details about helper functions.

The following is an example of recipe definitions in cookbook.

Kookbook.py: Compile hello.c so that to generate 'hello' command.

# product "hello" depends on "hello.o".
@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    """generates hello command"""           # recipe description
    system("gcc -g -o hello hello.o")

# product "hello.o" depends on "hello.c" and "hello.h".
@recipe
@product("hello.o")
@ingreds("hello.c", "hello.h")
def file_hello_o(c):
    """compile 'hello.c' and 'hello.h'"""   # recipe description
    system("gcc -g -c hello.c")

pyKook also provides short-notation. See the following example which is equivarent to the above, or see this section for details.

Example of Short-notation

# product "hello" depends on "hello.o".
@recipe("hello", ["hello.o"])
def file_hello(c):
    """generates hello command"""           # recipe description
    system("gcc -g -o hello hello.o")

# product "hello.o" depends on "hello.c" and "hello.h".
@recipe("hello.o", ["hello.c", "hello.h"])
def file_hello_o(c):
    """compile 'hello.c' and 'hello.h'"""   # recipe description
    system("gcc -g -c hello.c")

The following is an example of invoking pykook command.

• Command-line option '-l' shows recipes which have description. It means that recipes which have description are regarded as public recipes.
• Command-line option '-L' shows all recipes.

command-line example

bash> pykook -l
Properties:

Task recipes:

File recipes:
  hello               : generates hello command
  hello.o             : compile 'hello.c' and 'hello.h'

(Tips: you can set 'kookbook.default="XXX"' in your kookbook.)

bash> pykook hello
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

pyKook also provides kk command which is equivarent to pykook, because pykook is too long to type many times :) See this section for details.

Timestamp and content

pyKook checks both timestamp and content of files (= products, ingredients).

• If product is older than ingredients, that recipe will be executed.
• If product is newer than or have the same timestamp as ingredients, that recipe will not be executed.
• If recipe of ingredient is executed but content of ingredient is not changed, then recipe of product will not be executed and product will be 'touched'.
• If you specify command-line option '-F', these rules are ignored and all recipes are executed forcedly.

bash> rm -f hello hello.o
bash> kk hello                 ## 1st time
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

bash> kk hello                 ## 2nd time
                               ## nothing, because hello is already created.

bash> touch hello.c            ## touch hello.c
bash> kk hello                 ## 3rd time
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c            ## compiled, because hello.c is newer than hello.o.
### * hello (recipe=file_hello)
$ touch hello   # skipped       ## skipped, because content of hello.o is not changed.

bash> kk -F hello              ## 4th time (forcedly)
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

Product and Ingredients

Product and ingredient names are referable as property of recipe method's argument.

• c.product : product
• c.ingreds : ingredients
• c.ingred : same as c.ingreds[0]

Kookbook.py: Use c.product and c.ingreds

# product "hello" depends on "hello.o".
@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    """generates hello command"""
    system("gcc -g -o %s %s" % (c.product, c.ingred))
    # or system("gcc -g -o %s %s" % (c.product, c.ingreds[0]))
    # or system(c%"gcc -g -o $(product) $(ingreds[0])")

# product "hello.o" depends on "hello.c" and "hello.h".
@recipe
@product("hello.o")
@ingreds("hello.c", "hello.h")
def file_hello_o(c):
    """compile 'hello.c' and 'hello.h'"""
    system("gcc -g -c %s" % c.ingred)
    # or system("gcc -g -c %s" % c.ingreds[0])
    # or system(c%"gcc -g -c $(ingred)")

command-line example

bash> kk hello
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

pyKook provides convenience way to embed variables into string literal. For example, the followings are equivarent.

system("gcc -g -o %s %s" % (c.product, c.ingred))    # or c.ingreds[0]
system(c%"gcc -g -o $(product) $(ingred)")           # or $(ingreds[0])

You can write local or global variables in $() as well as product or ingreds.

CC     = 'gcc'             # global variable

@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    CFLAGS = '-g -Wall'    # local variable
    system(c%"$(CC) $(CFLAGS) -o $(product) $(ingreds[0])")

Specific recipe and generic recipe

Specific recipe is a recipe which is combined to a certain file. Product name of specific recipe is a concrete file name.

Generic recipe is a recipe which is combined to a pattern of file name. Product name of generic recipe is a pattern with metacharacter or regular expression.

pyKook converts file name pattern into regular expression. For example:

• '*.o' will be coverted into r'^(.*?)\.o$'.
• '*.??.txt' will be converted into to r'^(.*?)\.(..)\.txt$'.

Matched strings with metacharacter ('*' or '?') are accessable by $(1), $(2), ... in @ingreds() decorator.

Kookbook.py: Compile hello.c so that to generate 'hello' command.

## specific recipe
@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    """generates hello command"""
    system(c%"gcc -g -o $(product) $(ingred)")
    # or system("gcc -g -o %s %s" % (c.product, c.ingred))
    # or system("gcc -g -o %s %s" % (c.product, c.ingreds[0]))

## generic recipe
@recipe
@product("*.o")        # or @product(re.compile(r'^(.*?)\.o$'))
@ingreds("$(1).c", "$(1).h")
def file_ext_o(c):
    """compile '*.c' and '*.h'"""
    system(c%"gcc -g -c $(1).c")
    # or system("gcc -g -c %s.c" % c.m[1])
    # or system("gcc -g -c %s" % c.ingred)

command-line example

bash> kk -l
Properties:

Task recipes:

File recipes:
  hello               : generates hello command
  *.o                 : compile '*.c' and '*.h'

(Tips: you can set 'kookbook.default="XXX"' in your kookbook.)

bash> kk hello
### ** hello.o (recipe=file_ext_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

It is able to specify regular expression instead of filename pattern. For example, @product(re.compile(r'^(.*)\.o$')) is available as product instead of @product('*.o'). Grouping in regular expression is referable by $(1), $(2), ... in the same way.

Specific recipe is prior to generic recipe. For example, recipe 'hello.o' is used and recipe '*.o' is not used to generate 'hello.o' when target product is 'hello.o' in the following example.

Specific recipe is prior to generic recipe.

## When target is 'hello.o', this specific recipe will be used.
@recipe("hello.o", ["hello.c"])
def file_hello_o(c):
    system(c%"gcc -g -O2 -o $(product) $(ingred)")

## This generic recipe will not be used, because specific recipe
## is prior than generic recipe.
@recipe("*.o", ["$(1).c", "$(1).h"])
def file_o(c):
    system(c%"gcc -g     -o $(product) $(ingred)")

Conditional Ingredients

There may be a case that ingredient file exists or not. For example, product 'foo.o' depends on 'foo.c' and 'foo.h', while product 'bar.o' depends only on 'bar.c'.

In this case, you can use if_exists() helper function which resolve the problem. For example, when if_exists("hello.h") is specified in @ingreds(), pyKook detect dependency as following.

• If file 'hello.h' exists, product 'hello.o' depends on ingredients 'hello.c' and 'hello.h'.
• If file 'hello.h' doesn't exist, product 'hello.o' depends on only 'hello.c'.

if_exists() is useful especially when used with generic recipes.

Kookbook.py: Example of if_exists()

## specific recipe
@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    """generates hello command"""
    system(c%"gcc -g -o $(product) $(ingred)")
    # or system("gcc -g -o %s %s" % (c.product, c.ingred))
    # or system("gcc -g -o %s %s" % (c.product, c.ingreds[0]))

## generic recipe
@recipe
@product("*.o")        # or @product(re.compile(r'^(.*?)\.o$'))
@ingreds("$(1).c", if_exists("$(1).h"))
def file_hello_o(c):
    """compile '*.c' and '*.h'"""
    system(c%"gcc -g -c $(1).c")
    # or system("gcc -g -c %s.c" % c.m[1])
    # or system("gcc -g -c %s" % c.ingred)

command-line example

bash> kk hello
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o

File Recipe and Task Recipe

In pyKook, there are two kind of recipe.

File recipe

File recipe is a recipe which generates a file. In the other word, product of recipe is a file. If product is not generated, recipe execution will be failed.

Task recipe

Task recipe is a recipe which is not aimed to generate files. For example, task recipe 'clean' will remove '*.o' files and it doesn't generate any files.

Here is a matrix table of recipe kind.

	Specific recipe	Generic recipe
File recipe	Specific file recipe	Generic file recipe
Task recipe	Specific task recipe	Generic task recipe

pyKook determines recipe kind ('file' or 'task') according the following simple rule:

• File recipe should start with 'file_' prefix.
• Task recipe may stat with 'task_' prefix, or NOT decorated by @product().

In the following example, task recipe clean is a recipe to delete '*.o' files and is not combined to file 'clean'. Also task recipe all is a recipe to call recipe of 'hello' and is not combined to file 'all'.

Kookbook.py: Task recipes

## file recipe
@recipe
@product("hello")
@ingreds("hello.o")
def file_hello(c):
    """generates hello command"""
    system(c%"gcc -g -o $(product) $(ingred)")
    # or system("gcc -g -o %s %s" % (c.product, c.ingred))
    # or system("gcc -g -o %s %s" % (c.product, c.ingreds[0]))

## file recipe
@recipe
@product("*.o")        # or @product(re.compile(r'^(.*?)\.o$'))
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    """compile '*.c' and '*.h'"""
    system(c%"gcc -g -c $(1).c")
    # or system("gcc -g -c %s.c" % c.m[1])
    # or system("gcc -g -c %s" % c.ingred)

## task recipe
@recipe
def clean(c):
    """remove '*.o' files"""
    rm_f("*.o")

## task recipe
## (in order to avoid to overwrite 'all()' built-in function,
##  add 'task_' prefix to function name.)
@recipe
@ingreds("hello")
def task_all(c):
    """create all files"""
    pass

'pykook -l' will display task recipes and file recipes.

command-line example

bash> kk -l
Properties:

Task recipes:
  clean               : remove '*.o' files
  all                 : create all files

File recipes:
  hello               : generates hello command
  *.o                 : compile '*.c' and '*.h'

(Tips: you can set 'kookbook.default="XXX"' in your kookbook.)

bash> kk all
### *** hello.o (recipe=file_ext_o)
$ gcc -g -c hello.c
### ** hello (recipe=file_hello)
$ gcc -g -o hello hello.o
### * all (recipe=task_all)

bash> kk clean
### * clean (recipe=clean)
$ rm -f *.o

bash> ls -FC
Kookbook.py     hello*          hello.c         hello.h         optparse.c

pyKook have several well-known task name. Task recipes which product name is in the following list will be got pubilic automatically. For example, if you have defined 'all' task recipe, it will be displayed by 'kk -l' even when recicpe function doesn't have any description.

all

create all products

clean

remove by-products

clear

remove all products and by-products

deploy

deploy products

config

configure

setup

setup

install

install products

test

do test

Default Product

If you set product name into kookbook.default, pykook command will use it as default product.

Kookbook.py: specify default product name

## global variables
basename = 'hello'
command  = basename
kookbook.default = 'all'     # default product name

## file recipe
@recipe
@product(command)
@ingreds(basename + ".o")
def file_hello(c):
    """generates hello command"""
    system(c%"gcc -g -o $(product) $(ingred)")
    # or system("gcc -g -o %s %s" % (c.product, c.ingred))
    # or system("gcc -g -o %s %s" % (c.product, c.ingreds[0]))

## file recipe
@recipe
@product("*.o")        # or @product(re.compile(r'^(.*?)\.o$'))
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    """compile '*.c' and '*.h'"""
    system(c%"gcc -g -c $(1).c")
    # or system("gcc -g -c %s.c" % c.m[1])
    # or system("gcc -g -c %s" % c.ingred)

## task recipe
@recipe
def clean(c):
    """remove '*.o' files"""
    rm_f("*.o")

## task recipe
@recipe
@ingreds(command)
def task_all(c):
    """create all files"""
    pass

If you specify kookbook.default, you can omit target product name in commad-line.

command-line example

bash> kk           # you can omit target product name
### *** hello.o (recipe=file_ext_o)
$ gcc -g -c hello.c
### ** hello (recipe=file_hello)
$ gcc -g -o hello hello.o
### * all (recipe=task_all)

Properties

Property is a global variable which value can be overwrited in command-line option.

Property is defined by prop() function. It takes property name and default value as arguments.

Kookbook.py: properties

## global variables (not properties)
basename = 'hello'
kookbook.default = 'all'

## properties
CC       = prop('CC', 'gcc')
CFLAGS   = prop('CFLAGS', '-g -O2')
command  = prop('command', basename)

## file recipes
@recipe
@product(command)
@ingreds(basename + ".o")
def file_command(c):
    system(c%"$(CC) $(CFLAGS) -o $(product) $(ingred)")

@recipe
@product("*.o")
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    system(c%"$(CC) $(CFLAGS) -c $(ingred)")

## task recipes
@recipe
def clean(c):
    """remove '*.o' files"""
    rm_f("*.o")

@recipe
@ingreds(command)
def task_all(c):
    pass

Properties are shown when command-line option '-l' is specified.

command-line example

bash> kk -l
Properties:
  CC                  : 'gcc'
  CFLAGS              : '-g -O2'
  command             : 'hello'

Task recipes:
  clean               : remove '*.o' files
  all                 : cook all products

File recipes:

kookbook.default: all

(Tips: you can override properties with '--propname=propvalue'.)

If you don't specify any property values in command-line, default values are used.

command-line example

bash> kk all
### *** hello.o (recipe=file_ext_o)
$ gcc -g -O2 -c hello.c
### ** hello (recipe=file_command)
$ gcc -g -O2 -o hello hello.o
### * all (recipe=task_all)

If you specify property values in command-line, that values are used instead of default values.

bash> kk --command=foo --CFLAGS='-g -O2 -Wall' all
### *** hello.o (recipe=file_ext_o)
$ gcc -g -O2 -Wall -c hello.c
### ** foo (recipe=file_command)
$ gcc -g -O2 -Wall -o foo hello.o
### * all (recipe=task_all)

Property file is another way to specify properties. If you have create property file 'Properties.py' in current directory, pykook command reads it and set property values automatically.

Properties.py

CFLAGS = '-g -O2 -Wall'
command = 'foo'

Don't forget to write prop('prop-name', 'default-value') in your cookbook even when property file exists.

Result of pykook -l will be changed when property file exists.

bash> pykook -l
Properties:
  CC                  : 'gcc'
  CFLAGS              : '-g -O2 -Wall'
  command             : 'foo'

Task recipes:
  clean               : remove '*.o' files
  all                 : cook all products

File recipes:

kookbook.default: all

(Tips: you can override properties with '--propname=propvalue'.)

Materials

There is an exception in any case. Assume that you have a file 'optparse.o' which is supplied by other developer and no source. pyKook will try to find 'optparse.c' and failed in the result.

Using 'kookbook.materials', you can tell pyKook that 'optparse.o' is not a product.

Kookbook.py: materials

## global variables (not properties)
basename = 'hello'
kookbook.default = 'all'
kookbook.materials = ['optparse.o', ]   # specify materials

## properties
CC       = prop('CC', 'gcc')
CFLAGS   = prop('CFLAGS', '-g -O2')
command  = prop('command', basename)

## recipes
@recipe
@product(command)
@ingreds("hello.o", "optparse.o")
def file_command(c):
    system(c%"$(CC) $(CFLAGS) -o $(product) $(ingreds)")

@recipe
@product("*.o")
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    system(c%"$(CC) $(CFLAGS) -c $(ingred)")

@recipe
@ingreds(command)
def task_all(c):
    pass

In this example:

• 'hello.o' will be compiled from 'hello.c'.
• 'optparse.o' will not be compiled because it is specified as material.

command-line example

bash> kk all
### *** hello.o (recipe=file_ext_o)            ## only hello.o is compiled
$ gcc -g -O2 -c hello.c
### ** hello (recipe=file_command)
$ gcc -g -O2 -o hello hello.o optparse.o
### * all (recipe=task_all)

Command-line Options for Recipe

You can specify command-line options for certain recipes by @spices() decorator.

Kookbook.py: command-line options for recipes

## global variables (not properties)
basename = 'hello'
kookbook.default = 'all'
kookbook.materials = ['optparse.o', ]   # specify materials

## properties
CC       = prop('CC', 'gcc')
CFLAGS   = prop('CFLAGS', '-g -O2')
command  = prop('command', basename)

## recipes
@recipe
@product(command)
@ingreds("hello.o", "optparse.o")
def file_command(c):
    system(c%"$(CC) $(CFLAGS) -o $(product) $(ingreds)")

@recipe
@product("*.o")
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    system(c%"$(CC) $(CFLAGS) -c $(ingred)")

@recipe
@ingreds(command)
def all(c):
    pass

@recipe
@ingreds(command)
@spices("-d dir: directory to install (default '/usr/local/bin')",
        "--command=command: command name (default '%s')" % command)
def install(c, *args, **kwargs):
    opts, rests = kwargs, args
    dir = opts.get('d', '/usr/local/bin')  # get option value
    cmd = opts.get('command', command)     # get option value
    system(c%"sudo cp $(command) $(dir)/$(cmd)")   # or use 'install' command

Command-line options of recipes are displayed by '-l' or '-L' option.

command-line example

bash> kk -l
Properties:
  CC                  : 'gcc'
  CFLAGS              : '-g -O2'
  command             : 'hello'

Task recipes:
  all                 : cook all products
  install             : install product
    -d dir                directory to install (default '/usr/local/bin')
    --command=command     command name (default 'hello')

File recipes:

kookbook.default: all

(Tips: 'c%"gcc $(ingred)"' is more natural than '"gcc %s" % c.ingreds[0]'.)

You can specify command-line options for the recipe.

bash> kk install -d /tmp/local/bin --command=hellow
### * install (recipe=task_install)
$ sudo cp hello /tmp/local/bin/hellow
Password: *******

This feature can replace many small scripts with pyKook.

The following is an example to show styles of @spices arguments.

Kookbook.py: example of @spices()

@recipe
@spices("-h:      help",            # short opts (no argument)
        "-f file: filename",        # short opts (argument required)
        "-d[N]:   debug level",     # short opts (optional argument)
        "--help:  help",            # long opts (no argument)
        "--file=file: filename",    # long opts (argument required)
        "--debug[=N]: debug level", # long opts (optional argument)
        )
def echo(c, *args, **kwargs):
    """test of @spices"""
    opts, rests = kwargs, args
    print("opts: %r " % (opts,))
    print("rests: %r" % (rests,))

result

bash> kk -L
Properties:

Task recipes:
  echo                : test of @spices
    -h                    help
    -f file               filename
    -d[N]                 debug level
    --help                help
    --file=file           filename
    --debug[=N]           debug level

File recipes:

(Tips: you can override properties with '--propname=propvalue'.)

bash> kk echo -f hello.c -d99 --help --debug AAA BBB
### * echo (recipe=echo)
opts: {'debug': True, 'f': 'hello.c', 'help': True, 'd': 99}
rests: ('AAA', 'BBB')

Load Other Cookbooks

(Experimental)

It is possible to load other cookbooks by kookbook.load(). Using it, you can separate a large cookbook into several small cookbooks.

Common.py: common recipes and properties

## common properties
CC       = prop('CC', 'gcc')
CFLAGS   = prop('CFLAGS', '-g -O2')

## common recipes
@recipe
@product("*.o")
@ingreds("$(1).c", if_exists("$(1).h"))
def file_ext_o(c):
    """commpile *.c and *.h into *.o"""
    system(c%"$(CC) $(CFLAGS) -c $(ingred)")

Kookbook.py: load other cookbook

## global variables (not properties)
basename = 'hello'
kookbook.default = 'all'
kookbook.materials = ['optparse.o', ]   # specify materials

## properties
command  = prop('command', basename)

## load other cookbook
kookbook.load('./Common.py')             # or execfile('./Common.py')

## recipes
@recipe
@product(command)
@ingreds("hello.o", "optparse.o")
def file_command(c):
    system(c%"$(CC) $(CFLAGS) -o $(product) $(ingreds)")

@recipe
@ingreds(command)
def all(c):
    pass

@recipe
@ingreds(command)
@spices("-d dir: directory to install (default '/usr/local/bin')",
        "--command=command: command name (default '%s')" % command)
def install(c, *args, **kwargs):
    opts, rests = kwargs, args
    dir = opts.get('d', '/usr/local/bin')  # get option value
    cmd = opts.get('command', command)     # get option value
    system(c%"sudo cp $(command) $(dir)/$(cmd)")   # or use 'install' command

kookbook.load() accepts the following notations. Notice that these depend on __file__ of Kookbook.py, not on $PWD of current process.

Available format by kookbook.load()

kookbook.load('./book.py')          # load book in the same directory as 'Kookbook.py'
kookbook.load('../book.py')         # load book in parent directory of 'Kookbook.py'
kookbook.load('../../book.py')      # load book in parent's parent directory of 'Kookbook.py'
kookbook.load('.../book.py')        # search book in parent directory recursively
kookbook.load('~/book.py')          # load book in home directory
kookbook.load('@kook/tasks/clean.py')  # '@module' means os.path.dirname(module.__file__)

kookbook.load() imports recipes and properties, but not import other variables or functions. If you have variables or functions to be imported, specify their names to __export__.

Exports data or functions

__export__ = ('CLEAN_FILES', )

# this variable is exported, so user can add or manipulate
# filenames to be removed by 'clean' recipe.
CLEAN_FILES = ['**/*.pyc', '**/__pycache__']

@recipe
def clean(c):
    rm_rf CLEAN_FILES

If you want everything on other cookbook to be imported, use kookbook.load(bookname, True).

import other cookbook with context shared

kookbook.load('Common.py', True)

Other features

Category

Category is a class provided by Kook. It works as namespace.

Kookbook.py

from kook.utils import CommandOptionError

class git(Category):

  @recipe
  def default(c):
    """show status of working directory"""
    system("git status")

  @recipe
  @spices("-m MESSAGE: commit message")
  def ci(c):
    """commit current editing"""
    system("git commit -a")

  class branch(Category):

    @recipe
    def default(c):
      """show all branches"""
      system("git branch -a")

    @recipe
    def switch(c, *args):
      """switch current branch"""
      if not args:
        raise CommandOptionError("branch name is required.")
      system(c%"git co $(args[0])")

  class stash(Category):

    @recipe
    def default(c):
      """show all stashes"""
      system("git stash list")

    @recipe
    def save(c):
      """save stash with current date"""
      system("git stash save `date`")

    @recipe
    def pop(c):
      """pop the latest stash"""
      system("git stash pop")

Points:

• Define a subclass of 'Category' class.
• Define methods with '@recipe', and don't use self! These methods are regarded as just functions, not instance methods of defined class.
• Only task recipe is available in category. In other words, don't define file recipe in category.
• Nested category names are joined with ":".

Result

bash> kk -l
Properties:

Task recipes:
  git                 : show status of working directory
  git:ci              : commit current editing
    -m MESSAGE            commit message
  git:branch          : show all branches
  git:branch:switch   : switch current branch
  git:stash           : show all stashes
  git:stash:save      : save stash with current date
  git:stash:pop       : pop the latest stash

File recipes:

(Tips: you can set 'kookbook.default="XXX"' in your kookbook.)

NOTE:

Instance methods in category class are converted into staticmethods automatically.

class git(Category):
  @recipe
  def default(c):
    system("git status")

## methods in category are converted into staticmethods
assert type(git.__dict__['default']) == staticmethod
from types import FunctionType
assert type(git.default) == FunctionType

Therefore, you can call other recipe functions by category.method(), for example:

class apache(Category):

  @recipe
  def start(c):
    system("apachectl start")

  @recipe
  def stop(c):
    system("apachectl stop")

  @recipe
  def restart(c):
    apache.stop(c)
    apache.start(c)

Define Recipes Dinamically

You may define a lot of similar tasks.

Kookbook.py

class apache(Category):

    def task_start(c):
        """start apache process"""
        system("apachectl start")

    def task_stop(c):
        """stop apache process"""
        system("apachectl stop")

    def task_restart(c):
        """restart apache process"""
        system("apachectl restart")

In this case, you can define recipes dinamically by exec().

Kookbook.py

class apache(Category):

    for cmd in ('start', 'stop', 'restart'):
        code = r'''
@recipe
def task_%(cmd)s(c):
    """%(cmd)s apache process"""
    system("apachectl %(cmd)s")
''' % locals()
        exec(code)

Or you can define recipes by calling recipe() as function, not as decorator.

Kookbook.py

def def_task_recipe(command, ingreds_=(), spices_=()):
    @ingreds(*ingreds_)
    @spices(*spices_)
    def task_(c, *args, **kwargs):
        system("apachectl " + command)
    task_.__name__ = 'task_' + command
    task_.__doc__  = "%s apache process" % command
    return recipe(task_)

class apache(Category):
    for command in ('start', 'stop', 'restart'):
        fn = def_task_recipe(command, (), ())
        locals()[fn.__name__] = fn
    del fn

NOTE:

It is able to integrate these recipes into a recipe which can take arguments.

@recipe
@spices('command')
def task_apache(c, *args, **kwargs):
    """invoke apachectl (ex: kk apache start; kk apache -- -l)"""
    system("apachectl " + " ".join(args))

Meta Programming

(Experimental)

You can find and modify recipes as well as define recipes.

##
## Normaly, *.o is created from *.c.
##
@recipe('*.o', ['$(1).c'])
def file_o(c):
    """compile *.c into *.o"""
    system(c%"gcc -o $(ingred)")

##
## But you can change that rule for some files.
##
foo_recipe = kookbook['foo.o']
foo_recipe.ingreds.extend(('foo.h', 'bar.h'))
def func_foo_o(c):
    """generate foo.o from foo.c, foo.h and other.h"""
    ## invoke method of original recipe
    kookbook.get_recipe('*.o').method(c)    # same as file_o(c)
foo_recipe.method = func_foo_o
foo_recipe.desc = None     # make this recipe non-public

kookbook.find_recipe() is similar to kookbook[], but it doesn't register recipe automatically.

## For example:
kookbook['foo.o'].ingreds.append('foo.h')
## .. is same as:
r = kookbook.find_recipe('foo.o')
r.ingreds.append('foo.h')
kookbook.register(r)

Here is steps that kookbook[] and kookbook.find_recipe() do:

1. Find a specific recipe which matches to 'foo.o', but not found.
2. Then find a generic recipe, and a recipe '*.o' found.
3. Convert it into a specific recipe to suit 'foo.o'. for example:
   • Product: '*.o' => foo.o'
   • Ingreds: ['$(1).c'] => ['foo.c']
   • Method: (not changed)
4. Register it if kookbook[] called, but kookbook.find_recipe() doesn't.

Descrived as above, kookbook[] and kookbook.find_recipe() converts generic recipe into specific recipe. If you don't want to convert it, use kookbook.get_recipe().

@recipe('foo.o', ['foo.c', 'foo.h'])
def file_foo_o(c):
    ## invoke same command as *.o
    kookbook.get_recipe('*.o').method(c)

clean, sweep, and all recipes

Kook provides some useful recipes.

• Recipe clean is intended to remove by-products.
• Recipe sweep is intended to remove products and by-products.
• Recipe all is intended to produce all products.

Example of clean and sweep recipes

## load cookbook
## ('@kook' is equivarent to 'os.path.dirname(kook.__file__)')
kookbook.load("@kook/books/clean.py")               # 'clean' and 'swep' recipes

## add file patterns to remove
CLEAN.extend(["**/*.o", "**/*.class"])   # by-products
SWEEP.extend(["*.egg", "*.war"])         # products

#kookbook['sweep'].product = "clobber"              # if you like

Example of all recipe

## load cookbook
## ('@kook' is equivarent to 'os.path.dirname(kook.__file__)')
kookbook.load("@kook/books/all.py")      # 'all' recipe
## add product names you want to produce
ALL.extend(['product1', 'product2'])

Command-line Scripting Framework

pyKook supports to create command-line script.

The points are:

• Add '#!/usr/bin/env kk -X' as first line of script (shebang).
• Add 'kook_desc = "..script description.."'.
• Define specific task recipes which are regarded as sub-command.

'appsvr' script

#!/usr/bin/env pykook -X

from kook.utils import CommandOptionError

kook_desc = "start/stop web application server"

app = prop('app', 'helloworld')

@recipe
@spices("-p port: port number", "-d: debug")
def start(c, *args, **kwargs):
    """start server process"""
    p = kwargs.get("p", 8080)
    d = kwargs.get("d") and "-d" or ""
    _app = args and args[0] or app
    system("nohup python dev_appserver.py -p %s %s %s &" % (p, d, _app))

@recipe
def stop(c):
    """stop server process"""
    system_f("ps auxw | awk '/python dev_appserver.py/ && !/awk/{print $2}' | xargs kill")

result

### Don't forget to make script executable!
bash> chmod a+x appsvr

### Show help
bash> ./appsvr -h
appsvr - start/stop web application server

sub-commands:
  start           : start server process
  stop            : stop server process

(Type 'appsvr -h subcommand' to show options of sub-commands.)

### Show help for each sub-command
bash> ./appsvr -h start
appsvr start - start server process
  -p port              : port number
  -d                   : debug

### Invoke sub-command
bash> ./appsvr start -p 4223
appending output to nohup.out
bash> ./appsvr stop

Short command

pyKook provides kk command which is the same as pykook command, because pykook is too long to type many times :)

bash> kk all    # this is more confortable to type than pykook :)

In fact, kk is a shell script to invoke pykook or plkook command according to filename of cookbook. For example, pykook will be invoked by kk when Kookbook.py exists, or plkook will be invoked when Kookbook.pl exists. Therefore kk script requires Kookbook.py to invoke pykook command.

### you can't invoke kk when Kookbook.py doesn't exist
bash> ls Kookbook.py
ls: Kookbook.py: No such file or directory
bash> kk -h
kk: No kookbook found.

In addition, kk searches Kookbook.py in parent directory recursively.

bash> ls -F
Kookbook.py    src/    test/
bash> cd src/foo/bar/
bash> ls Kookbook.py
ls: Kookbook.py: No such file or directory
bash> kk clean                       # OK
### * clean (recipe=clean)
$ rm **/*.pyc

NOTE:

Notice that current directory will be changed to parent directory in which Kookbook.py exists.

Short notation

pyKook provides short notation of recipe.

### normal notation                      ### short notation
@recipe                                  @recipe('*.o', ['$(1).c', '$(1).h'])
@product('*.o')                          def file_o(c):
@ingreds('$(1).c', '$(1).h')                system(c%"gcc -o $(ingred)")
def file_o(c):
   system(c%"gcc -c $(ingred)")

@recipe                                  @recipe('build', ['hello.o'])
@ingreds('hello.o')                      def task_build(c):
def build(c):                               system(c%"gcc -o hello *.o")
   system(c%"gcc -o hello *.o")

@recipe() decorator can take two arguments.

• 1st argument represents product. If you pass None, it will be ignored.
• 2nd argument represents ingredients and should be list or tuple of string. And 2nd argument is optional.

Debug mode

Command-line option -D or -D2 turn on debug mode and debug message will be displayed. -D2 is higher debug level than -D.

example of -D

bash> kk -D hello
*** debug: + begin hello
*** debug: ++ begin hello.o
*** debug: +++ material hello.c
*** debug: +++ material hello.h
*** debug: ++ create hello.o (recipe=file_hello_o)
### ** hello.o (recipe=file_hello_o)
$ gcc -g -c hello.c
*** debug: ++ end hello.o (content changed)
*** debug: + create hello (recipe=file_hello)
### * hello (recipe=file_hello)
$ gcc -g -o hello hello.o
*** debug: + end hello (content changed)

Invoke Recipes Forcedly

Command-line option '-F' invokes recipes forcedly. In the other words, timestamp of files are ignored when '-F' is specified.

Nested Array

You can specify not only filenames but also list of filenames as ingredient @ingreds(). pyKook flatten arguments of @ingreds() automatically.

Kookbook.py: specify list of filenames

from glob import glob
sources = glob("*.c")
objects = [ s.replace(".c", ".o") for s in sources ]

@recipe
@product("hello")
@ingreds(objects)    ## specify list of filenams
def file_hello(c):
    system(c%"gcc -o $(product) $(ingreds)")  # expanded to filenames

@recipe
@product("*.o")
@ingreds("$(1).c")
def file_ext_o(c):
    sytem(c%"gcc -c $(ingred)")

Cookbook Concatenation

It is possible to concatenate your cookbook and pyKook libraries into a file. Using concatenated file, user can use your cookbook without installing pyKook.

To concatenate files, add the following into your Kookbook.py::

kookbook.load('@kook/books/concatenate.py')
#CONCATENATE_MODULES.append(foo.bar.module)  # if you want
#CONCATENATE_BOOKS.append('foo/bar/book.py') # if you want

And type the following commands in terminal::

bash> pykook concatenate -o yourscript.py Kookbook.py
bash> chmod a+x yourscript.py
bash> ./yourscript.py -l

If you don't specify Kookbook.py, it means that all pyKook libraries are concatenated into a file. You can use it instead of 'pykook' command.

bash> pykook concatenate -o yourscript   # 'yourscript' contains all pyKook library content
bash> chmod a+x yourscript
bash> ./yourscript -h        # 'yourscrit' can be an alternative of pykook command

Remote Task Recipe

pyKook allows you to define task recipes which runs commands on remote machine by ssh. This is very useful when you want to deploy your application to servers.

Before using remote task recipe, you must install pycrypto and paramiko.

Install pycrypto and paramiko

## install
bash> sudo pip install pycrypto
bash> sudo pip install paramiko

NOTE:

You may got error when installing pycrypto by easy_install.

bash> easy_install pycrypto
Searching for pycrypto
Reading http://pypi.python.org/simple/pycrypto/
Reading http://www.pycrypto.org/
Reading http://pycrypto.sourceforge.net
Reading http://www.amk.ca/python/code/crypto
Best match: pycrypto 2.4
Downloading https://ftp.dlitz.net/pub/dlitz/crypto/pycrypto/pycrypto-2.4.tar.gz
Processing pycrypto-2.4.tar.gz
Running pycrypto-2.4/setup.py -q bdist_egg --dist-dir /var/folders/FD/FDjI6Ce4H7eSxs5w+QNj+k+++TI/-Tmp-/easy_install-zEh6X7/pycrypto-2.4/egg-dist-tmp-qqBby0
error: Setup script exited with error: src/config.h: No such file or directory

To avoid it, I recommend you to install pycrypto by pip.

bash> easy_install pip
bash> pip install pycrypto

Ssh Configuration

You must finish ssh configuration before using pyKook's remote task recipe. If you already finished it, go to next section.

Ssh configuration example

### generate public/private RSA keys.
bash> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/yourname/.ssh/id_rsa): Enter
Enter passphrase (empty for no passphrase): Enter
Enter same passphrase again: Enter
Your identification has been saved in /home/yourname/.ssh/id_rsa.
Your public key has been saved in /home/yourname/.ssh/id_rsa.pub.
The key fingerprint is:
ab:cd:ef:12:34:56:78:90:ab:cd:ef:12:34:56:78:90 yourname@localhost
bash> ls ~/.ssh/id_rsa.pub
id_rsa.pub

### copy public key to server
bash> scp ~/.ssh/id_rsa.pub user1@server1:id_rsa.pub
Password: ********

### register public key
bash> ssh user1@server1
Password: ********
[server1]> ls id_rsa.pub
id_rsa.pub
[server1]> mkdir -p ~/.ssh
[server1]> cat id_rsa.pub >> ~/.ssh/authorized_keys
[server1]> rm id_rsa.pub
[server1]> chmod 600 ~/.ssh/authorized_keys
[server1]> chmod 700 ~/.ssh

### confirm that you can log-in to server without entering password
[server1]> exit
bash> ssh user1@server1
[server1]>

Remote Object

Using kook.remote.Remote object, you can define remote task recipe.

Kookbook.py: basic example to define remote object

from kook.remote import Remote

remote = Remote(
  hosts = ['dev.example.org'],
  port  = 10022,    # default 22
  user  = 'user1',
  #passphrase = 'XXXXXXXX',  # passphrase for ~/.ssh/id_rsa (if necessary)
)

## or
#remote = Remote(
#  hosts = ['user1@dev.example.org:10022'],
#)

class deploy(Category):

  @recipe
  @remotes(remote)
  def info(c):
    """show remote host information"""
    ssh = c.ssh
    ssh('hostname')
    ssh('whoami')

  ### or
  #@recipe(remotes=[remote])
  #def info(c):
  #  ....

Output Exampl

bash> kk deploy:info
### * deploy:info (recipe=info)
[user1@dev.example.org]$ hostname
dev
[user1@dev.example.org]$ whoami
user1

Of course you can define local and remote task recipes in a cookbook. In addition you can mix local and remote command in a recipe.

Mix local and remote command in a recipe

@recipe
@rempte
def info(c):
  ssh = c.ssh
  ssh('hostname')      # on remote
  system('hostname')   # on local

If you want to define some roles of remote hosts, create remote objects for each role.

Create remote objects for each role

remote_web = Remote(
  hosts = ['www1.example.org', 'www2.example.org'],
  user  = 'wwwadmin',
)
remote_db = Remote(
  hosts = ['db.example.com'],
  user  = 'dbadmin',
)

class web(Category):
  @recipe
  @remotes(remote_web)
  def restart(c):
    ...

class db(Category):
  @recipe
  @remotes(remote_db)
  def restart(c):
    ...

Available Commands

The following methods are available on c.ssh or c.sftp:

ssh(OS-command)

Same as ssh.system(OS-command).

## run 'pwd' command on remote machne
ssh('pwd')

ssh.system(OS-command)

Runs OS-command on remote machine. Raises exception when command failed.

## run 'pwd' command on remote machne
ssh.system('pwd')

ssh.system_f(OS-command)

Runs OS-command on remote machine, and ignore status.

## run 'cat' command with non-exist file
ssh.system_f('cat not-exist.txt')

ssh.sudo(OS-command)

Runs OS-command by sudo on remote machine.

## run 'sudo make install' command
ssh.sudo('make install')

See the next section.

ssh.cd(remote-directory)

Change remote directory.

## change directory on remote machine
ssh.cd('app/repo')

ssh.pushd(remote-directory)

Using 'with' statement, enter and exit remote directory.

ssh("pwd")        #=> ex. /home/admin
with ssh.pushd("app/repo"):
    ssh("pwd")    #=> ex. /home/admin/app/repo
ssh("pwd")        #=> ex. /home/admin

ssh.getcwd()

Returns current directory on remote machine.

## print current directory on remote machine
print(getcwd())

sftp.listdir(path)

Returns list of filenames of path on remote machine. If path doesn't exist on remote machine then throws exception.

## lists filenames of current directory
filenames = sftp.listdir('.')

sftp.listdir_f(path)

Returns list of filenames of path on remote machine. Returns an empty list even when path doesn't exist.

## create log directory only when not exist
filenames = ssh.listdir_f('var/log')
if not filenames:
    ssh('mkdir -p var/log')

sftp.get(remote-filename [, local-filename])

Downloads remote file to local machine.

## download 'logo.png'
ssh.get('logo.png')
## download 'logo.png' as 'logo_20110101.png'
ssh.get('logo.png', 'logo_20110101.png')

sftp.put(local-filename [, remote-filename])

Uploads local file to remote machine.

## upload 'logo.png'
ssh.put('logo.png')
## upload 'logo_20110101.png' as 'logo.png'
ssh.get('logo_20110101.png', 'logo.png')

sftp.mget(remote-filename1, remote-filename2, ...)

Downloads remote files to local machine. Glob pattern (such as '*.html') is available.

## download image files
ssh.mget('*.jpg', '*.png', '*.gif')

sftp.mput(local-filename1, local-filename2, ...)

Upload local files to remote machine. Glob pattern (such as '*.html') is available.

## upload image files
ssh.mput('*.jpg', '*.png', '*.gif')

Current directory on remote machine is shared between c.ssh and c.sftp.

@recipe
@remotes(remote)
def upload(c):
    ssh, sftp = c.ssh, c.sftp
    cd("img")
    ## upload image files to 'img/' on remote machine
    with ssh.pushd("img"):
        sftp.mput("*.png", "*.jpg", "*.gif")

Sudo Command

Kookbook.py: basic example to define remote object

from kook.remote import Remote

remote = Remote(
  hosts = ['dev.example.org'],
  port  = 10022,    # default 22
  user  = 'user1',
  password      = 'XXXXXXXX',   # for login, ~/.ssh/id_rsa, and sudo command
  #passphrase    = 'XXXXXXXX',  # only for ~/.ssh/id_rsa
  #sudo_password = 'XXXXXXXX',  # only for sudo command
)

class deploy(Category):

  @recipe
  @remotes(remote)
  def info(c):
    """show remote host information"""
    ssh = c.ssh
    ssh('hostname')
    ssh('whoami')
    ssh.sudo('whoami')

Output Exampl

bash> kk deploy:info
### * deploy:info (recipe=info)
[user1@dev.example.org]$ hostname
dev
[user1@dev.example.org]$ whoami
user1
[user1@dev.example.org]$ sudo whoami
root

NOTE:

If you got the following error, add 'Defaults visiblepw' into '/etc/sudoers' on remote server.

[user1@dev.example.org]$ sudo whoami
*** ERROR
pykook: sudo: no tty present and no askpass program specified
 (Hint: add 'Defaults visiblepw' into '/etc/sudoers' with 'visudo' command)
  File "/usr/local/lib/python2.7/site-packages/kook/main.py", line 202, in main
    status = self.invoke()
...(snip)...
  File "/usr/local/lib/python2.7/site-packages/kook/remote.py", line 362, in _check_sudo_password
    raise KookCommandError(self._add_hint_about_sudo_settings(errmsg))

Password Object

If you don't want to embed password into cookbook, use kook.remote.Password object.

Kookbook.py: Password object example

from kook.remote import Remote, Password

remote = Remote(
  hosts = ['dev.example.org'],
  port  = 10022,    # default 22
  user  = 'user1',
  password      = Password('login'),
  #passphrase    = Passowrd('~/.ssh/id_rsa'),
  #sudo_password = Passowrd('sudo command'),
)

class deploy(Category):

  @recipe
  @remotes(remote)
  def info(c):
    """show remote host information"""
    ssh = c.ssh
    ssh('hostname')
    ssh('whoami')
    ssh.sudo('whoami')

Output Exampl

bash> kk deploy:info
### * deploy:info (recipe=info)
Password: ********
[user1@dev.example.org]$ hostname
dev
[user1@dev.example.org]$ whoami
user1
[user1@dev.example.org]$ sudo whoami
root

Using password object, it is easy to share a password. For example:

Share password object

passwd = Password()

app_servers = Remote(
  hosts = ['www1.example.org', 'www2.example.org']
  user  = 'user1',
  password      = passwd,
  #passphrase    = passwd,
  #sudo_password = passwd,
)

db_servers = Remote(
  hosts = ['db1.example.com']
  user  = 'user2',
  password      = passwd,
  #passphrase    = passwd,
  #sudo_password = passwd,
)

In above example, you will asked password only once, because a password object is shared between remote objects.

Switching Remote Hosts

If you want to switch hosts between production and staging environment, property will help you.

Kookbook.py: Switching production and staging servers

from kook.remote import Remote, Password

production = prop('production', False)
if production:
  hosts = ['www.example.com']     ## production server
else:
  hosts = ['dev.example.com']     ## staging server

remote = Remote(
  hosts = hosts,
  port  = 10022,    # default 22
  user  = 'user1',
  password = Password('login'),
)

class deploy(Category):

  @recipe
  @remotes(remote)
  def info(c):
    """show remote host information"""
    ssh = c.ssh
    ssh('hostname')

Output Example

bash> kk --production deploy:info
### * deploy:info (recipe=info)
[user1@www.example.org]$ hostname
www.example.org

Deployment Example

Here is an example of remote task recipe for deployment.

from __future__ import with_statement
from kook.remote import Remote, Password

## repository url
repository_url = "git@bitbucket.org:yourname/myapp1.git"

## production or staging servers
production = prop('production', False)
if production:
    hosts = ['www.example.com']     ## production server
else:
    hosts = ['dev.example.com']     ## staging server

app_server = Remote(
    hosts = hosts,
    port  = 10022,    # default 22
    user  = 'user1',
    password = Password(),
)

## recipe definitions
class deploy(Category):

    @recipe
    @remotes(app_server)
    @spices('-t tag: tag name to checkout')
    def default(c, *args, **kwargs):
        """deploy to remote server"""
        tagname = kwargs.get('t')
        ssh = c.ssh
        if "repo" not in ssh.listdir_f("app"):
            ssh("mkdir -p app/repo")
        ## call other recipe to check-out source code
        deploy.checkout(c, *args, **kwargs)
        ## deploy
        target = tagname or 'master'
        ssh(c%"mkdir -p app/releases/$(target)")
        with ssh.cd(c%"app/releases/$(target)"):
            ## copy source files
            ssh("cp -pr ../../repo/* .")
            ## migrate database by 'migrate'
            #ssh("python db_repository/manage.py upgrade")
        ## recreate symbolic link
        with ssh.cd("app/releases"):
            ssh(c%"rm -f current")
            ssh(c%"ln -s $(tagname) current")

    @recipe
    @remotes(app_server)
    @spices('-t tag: tag name to checkout')
    def checkout(c, *args, **kwargs):
        """checkout source code from git repository"""
        tagname = kwargs.get('t')
        ssh = c.ssh
        if "repo" not in ssh.listdir_f("app"):
            ssh("mkdir -p app/repo")
        ## checkout git repository
        with ssh.cd("app/repo"):
            files = ssh.listdir(".")
            if '.git' not in files:
                ssh(c%"git clone $(repository_url) .")
            else:
                ssh(c%"git fetch")
            if tagname:
                ssh(c%"git checkout -q refs/tags/$(tagname)")
            else:
                ssh(c%"git checkout master")

Restrictions

(Experimental)

There are some restrictions about remote task recipe.

• Remote task recipes are not executed in parallel. Therefore if you try to deploy your application into thousands remote machine, pyKook will take alot of time.

• It is not recommended to specify remote recipes as ingredients of remote recipe.

Kookbook.py

        class apache(Category):
          @recipe(remotes=[remote])
          def start(c): ...

          @recipe(remotes=[remote])
          def start(c): ...

          #@recipe(remotes=[remote])
          #@ingreds('start', 'stop')    # not recommended!
          #def restart(c):
          #  pass
          @recipe(remotes=[remote])
          def restart(c):
            apache.start(c)        # execute remote recipe instead
            apache.stop(c)         # execute remote recipe instead

Because dependency between remote recipes is not solved as what you expected.

Kookbook.py

from kook.remote import Remote
remote = Remote(hosts=['host1', 'host2', 'host3'])
#
@recipe
@remotes(remote)
def prepare(c):
    print('prepare: ' + c.session.host)
#
@recipe
@remotes(remote)
@ingreds('prepare')
def maintask(c):
    print('maintask: ' + c.session.host)

Output example

bash> kk maintask
### ** prepare (recipe=prepare)
prepare: host1
prepare: host2
prepare: host3
### * maintask (recipe=maintask)
maintask: host1
maintask: host2
maintask: host3

Trouble Shooting

xxx: product not created (in file_xxx())

Q: I got the "xxx: product not created (in file_xxx())." error.

A: You may define file recipe instead of task recipe. Don't specify '@product()' if you want to define task recipe.

    ## This will cause error
    @recipe
    @product("clean")
    def clean(c):   #=> KookRecipeError: clean: product not created (in file_clean()).
        rm_f("*.o")

    ## Don't specify @product()
    @recipe
    def clean(c):   #=> ok
        rm_f("*.o")

    ## Or add 'task_' prefix to function name
    @recipe
    @product("clean")
    def task_clean(c):        #=> almost equivarent to above recipe
        rm_f("*.o")

*.c: can't find any recipe to produce.

Q: I got the "*.c: can't find any recipe to produce." error.

A: Use "$(1).c" instead of "*.c" in @ingreds() argument.

    ## This will cause error because "*.c" is used in ingredients.
    @recipe
    @product("*.o")
    @ingreds("*.c")  #=> KookRecipeError: *.c: can't find any recipe to produce.
    def file_ext_o(c):
        system(c%"gcc -c $(ingred)")

    ## Use "$(1).c" instead of "*.c"
    @recipe
    @product("*.o")
    @ingreds("$(1).c")  #=> ok
    def file_ext_o(c):
        system(c%"gcc -c $(ingred)")

sh: line 1: ingred: command not found

Q: I got the "sh: line 1: ingred: command not found" error.

A: Add "c%" at the beginning of command string.

    ## "c%" is forgetten
    @recipe
    @product("*.o")
    @ingreds("$(1).c")
    def file_ext_o(c):
        system("gcc -c $(ingred)")
            #=> KookCommandError: sh: line 1: ingred: command not found" error.

    ## Don't forget to add "c%" if you want to use "$()".
    @recipe
    @product("*.o")
    @ingreds("$(1).c")
    def file_ext_o(c):
        system(c%"gcc -c $(ingred)")

References

Filesystem Functions

The following functions are available in recipe.

system(cmmand-string)

Execute command-string. If command status is not zero then exception is raised.

system("gcc hello.c")

system_f(command-string)

Execute command-string. Command statuis is ignored.

echo(string)

Echo string. Newline is printed.

echo("OK.")

echo_n(string)

Echo string. Newline is not printed.

echo_n("Enter your name: ")

cd(dir)

Change directory. Return current directory.

cwd = cd("build")
...
cd(cwd)              # back to current directry

chdir(dir, callable=None)

Change current directory temporary. If this is used with Python's with-statement, current directory will be backed automatically.

with chdir('test') as d:
    ## in 'test' directory
    system('python test_all.py')
## back to current directry automatically

Or, if you are using Python 2.4 or older, callable object is available as 2nd argument in order the same purpose as with-statemnet.

def f():
    sytem('python test_all.py')
chdir('test', f)   ## f() is called at 'test' directory
## back to current directry automatically

pushd(dir)

Change current directory temporary. If this is used with Python's with-statement, current directory will be backed automatically.

with pushd('test') as d:
    ## in 'test' directory
    system('python test_all.py')
## back to current directry automatically

Or it is available as function decorator. This is useful in Python 2.4 or older which doesn't support 'with' statement.

@pushd('test')
def do():
    sytem('python test_all.py')  ## invoked on 'test' directory
## back to current directry automatically

mkdir(path)

Make directory.

mkdir("lib")

mkdir_p(path)

Make directory. If parent directory is not exist then it is created automatically.

mkdir_p("foo/bar/baz")

rm(path[, path2, ...])

Remove files.

rm('*.html', '*.txt')

rm_r(path[, path2, ...])

Remove files or directories recursively.

rm_r('*')

rm_f(path[, path2, ...])

Remove files forcedly. No errors reported even if path doesn't exist.

rm_f('*.html', '*.txt')

rm_rf(path[, path2, ...])

Remove files or directories forcedly. No errors reported even if path doesn't exist.

rm_rf('*')

touch(path[, path2, ...])

Touch files or directories. If path doesn't exist then empty file is created.

touch('*.c')

cp(file1, file2)

Copy file1 to file2.

cp('foo.txt', 'bar.txt')

cp(file, file2, ..., dir)

Copy file to dir.

cp('*.txt', '*.html', 'dir')

cp_r(path1, path2)

Copy path1 to path2 recursively.

cp_r('dir1', 'dir2')

cp_r(path, path2, ..., dir)

Copy path to dir recursively. Directory dir must exist.

cp_r('lib', 'doc', 'test', 'dir')

cp_p(file1, file2)

Copy file1 to file2. Timestams is preserved.

cp_p('foo.txt', 'bar.txt')

cp_p(file, file2, ..., dir)

Copy file to dir. Timestams is preserved. Directory dir must exist.

cp_p('*.txt', '*.html', 'dir')

cp_pr(path1, path2)

Copy path1 to path2 recursively. Timestams is preserved.

cp_pr('lib', 'lib.bkup')

cp_pr(path, path2, ..., dir)

Copy path to dir recursively. Directory dir must exist. Timestams is preserved.

cp_pr('lib/**/*.rb', 'test/**/*.rb', 'tmpdir')

mv(file1, file2)

Rename file1 to file2.

mv('foo.txt', 'bar.txt')

mv(path, path2, ..., dir)

Move path to dir.

mv('lib/*.rb', 'test/*.rb', 'tmpdir')

store(path, path2, ..., dir)

Copy path (files or directories) to dir with keeping path-structure.

store("lib/**/*.py", "doc/**/*.{html,css}", "dir")
## ex.
##   "lib/kook/__init__.py"  is copied into "dir/lib/kook/__init__.py"
##   "lib/kook/utils.py"     is copied into "dir/lib/kook/utils.py"
##   "lib/kook/main.py"      is copied into "dir/lib/kook/main.py"
##   "doc/users-guide.html"  is copied into "dir/doc/users-guide.html"
##   "doc/docstyle.css"      is copied into "dir/doc/docstyle.css"

store_p(path, path2, ..., dir)

Copy path (files or directories) to dir with keeping path-structure. Timestamp is preserved.

store_p("lib/**/*.py", "doc/**/*.html", "dir")

edit(path, path2, ..., by=replacer)

Edit file content. Keyword argument 'by' should be a callable to edit content, or list of tuples of replacing pattern and string.

## edit by list of regular expression and string
replacer = [
    (r'\$Release\$', "1.0.0"),
    (r'\$Copyright\$', "copyright(c) 2008 kuwata-lab.com"),
]
edit("lib/**/*.py", "doc/**/*.{html,css}", by=replacer)
## edit by function
def replacer(s):
    s = s.replace('0.7.2',   "1.0.0", s)
    s = s.replace('copyright(c) 2008-2012 kuwata-lab.com all rights reserved.', "copyright(c) 2008 kuwata-lab.com", s)
    return s
edit("lib/**/*.py", "doc/**/*.{html,css}", by=replacer)

The above functions can take lists or tuples as file or directory names. (If argument is list or tuple, it is flatten by kook.utils.flatten().)

For example, the following code is available.

## copy all files into dir
files = ['file1.txt', 'file2.txt', 'file3.txt']
cp(files, 'dir')

The following file pattern is available.

*

Matches sequence of any character.

?

Matches a character.

{a,b,c}

Matches a or b or c.

**/

Matches directory recursively.