doc: Add pk debugging section.

[8sync.git] / doc / 8sync-new-manual.org

# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled ``GNU
# Free Documentation License''.
# 
# A copy of the license is also available from the Free Software
# Foundation Web site at http://www.gnu.org/licenses/fdl.html
# 
# Altenately, this document is also available under the Lesser General
# Public License, version 3 or later, as published by the Free Software
# Foundation.
# 
# A copy of the license is also available from the Free Software
# Foundation Web site at http://www.gnu.org/licenses/lgpl.html

* Preface

Welcome to 8sync's documentation!
8sync is an asynchronous programming environment for GNU Guile.
(Get it? 8sync? Async??? Quiet your groans, it's a great name!)

8sync has some nice properties:

 - 8sync uses the actor model as its fundamental concurrency
   synchronization mechanism.
   Since the actor model is a "shared nothing" asynchronous
   environment, you don't need to worry about deadlocks or other
   tricky problems common to other asynchronous models.
   Actors are modular units of code and state which communicate
   by sending messages to each other.
 - If you've done enough asynchronous programming, you're probably
   familiar with the dreaded term "callback hell".
   Getting around callback hell usually involves a tradeoff of other,
   still rather difficult to wrap your brain around programming
   patterns.
   8sync uses some clever tricks involving "delimited continuations"
   under the hood to make the code you write look familiar and
   straightforward.
   When you need to send a request to another actor and get some
   information back from it without blocking, there's no need
   to write a separate procedure... 8sync's scheduler will suspend
   your procedure and wake it back up when a response is ready.
 - Even nonblocking I/O code is straightforward to write.
   Thanks to the "suspendable ports" code introduced in Guile 2.2,
   writing asynchronous, nonblocking networked code looks mostly
   like writing the same synchronous code.
   8sync's scheduler handles suspending and resuming networked
   code that would otherwise block.
 - 8sync aims to be "batteries included".
   Useful subsystems for IRC bots, HTTP servers, and so on are
   included out of the box.
 - 8sync prioritizes live hacking.
   If using an editor like Emacs with a nice mode like Geiser,
   an 8sync-using developer can change and fine-tune the behavior
   of code /while it runs/.
   This makes both debugging and development much more natural,
   allowing the right designs to evolve under your fingertips.
   A productive hacker is a happy hacker, after all!

In the future, 8sync will also provide the ability to spawn and
communicate with actors on different threads, processes, and machines,
with most code running the same as if actors were running in the same
execution environment.

But as a caution, 8sync is still very young.
The API is stabilizing, but not yet stable, and it is not yet well
"battle-tested".
Hacker beware!
But, consider this as much an opportunity as a warning.
8sync is in a state where there is much room for feedback and
contributions.
Your help wanted!

And now, into the wild, beautiful frontier.
Onward!

* Tutorial

** Intro to the tutorial

IRC!  Internet Relay Chat!
The classic chat protocol of the Internet.
And it turns out, one of the best places to learn about networked
programming.

In the 1990s I remember stumbling into some funky IRC chat rooms and
being astounded that people there had what they called "bots" hanging
around.
From then until now, I've always enjoyed encountering bots whose range
of functionality has spanned from saying absurd things, to taking
messages when their "owners" were offline, to reporting the weather,
to logging meetings for participants.
And it turns out, IRC bots are a great way to cut your teeth on
networked programming; since IRC is a fairly simple line-delineated
protocol, it's a great way to learn to interact with sockets.
(My first IRC bot helped my team pick a place to go to lunch, previously
a source of significant dispute!)
At the time of writing, venture capital awash startups are trying to
turn chatbots into "big business"... a strange (and perhaps absurd)
thing given chat bots being a fairly mundane novelty amongst hackers
and teenagers everywhere in the 1990s.

We ourselves are going to explore chat bots as a basis for getting our
feet wet in 8sync.
We'll start from a minimalist example using an irc bot with most of
the work done for us, then move on to constructing our own actors as
"game pieces" which interface with our bot, then experiment with just
how easy it is to add new networked layers by tacking on a high score
to our game, and as a finale we'll dive into writing our own little
irc bot framework "from scratch" on top of the 8sync actor model.

Alright, let's get started.
This should be a lot of fun!

** A silly little IRC bot

First of all, we're going to need to import some modules.  Put this at
the top of your file:

#+BEGIN_SRC scheme
  (use-modules (8sync)               ; 8sync's agenda and actors
               (8sync systems irc)   ; the irc bot subsystem
               (oop goops)           ; 8sync's actors use GOOPS
               (ice-9 format)        ; basic string formatting
               (ice-9 match))        ; pattern matching
#+END_SRC

Now we need to add our bot.  Initially, it won't do much.

#+BEGIN_SRC scheme
  (define-class <my-irc-bot> (<irc-bot>))

  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (if emote?
        (format #t "~a emoted ~s in channel ~a\n"
                speaker line channel)
        (format #t "~a said ~s in channel ~a\n"
                speaker line channel)))
#+END_SRC

We've just defined our own IRC bot!
This is an 8sync actor.
(8sync uses GOOPS to define actors.)
We extended the handle-line generic method, so this is the code that
will be called whenever the IRC bot "hears" anything.
For now the code is pretty basic: it just outputs whatever it "hears"
from a user in a channel to the current output port.
Pretty boring!
But it should help us make sure we have things working when we kick
things off.

Speaking of, even though we've defined our actor, it's not running
yet.  Time to fix that!

#+BEGIN_SRC scheme
(define* (run-bot #:key (username "examplebot")
                  (server "irc.freenode.net")
                  (channels '("##botchat")))
  (define hive (make-hive))
  (define irc-bot
    (hive-create-actor* hive <my-irc-bot> "irc-bot"
                        #:username username
                        #:server server
                        #:channels channels))
  (run-hive hive (list (bootstrap-message hive irc-bot 'init))))
#+END_SRC

Actors are connected to something called a "hive", which is a
special kind of actor that runs all the other actors.
Actors can spawn other actors, but before we start the hive we use
this special "hive-create-actor*" method.
It takes the hive as its first argument, the actor class as the second
argument, a decorative "cookie" as the third argument (this is
optional, but it helps with debugging... you can skip it by setting it
to #f if you prefer), and the rest are initialization arguments to the
actor.  hive-create-actor* passes back not the actor itself (we don't
get access to that usually) but the *id* of the actor.
(More on this later.)
Finally we run the hive with run-hive and pass it a list of
"bootstrapped" messages.
Normally actors send messages to each other (and sometimes themselves),
but we need to send a message or messages to start things or else
nothing is going to happen.

We can run it like:

#+BEGIN_SRC scheme
(run-bot #:username "some-bot-username") ; be creative!
#+END_SRC

Assuming all the tubes on the internet are properly connected, you
should be able to join the "##botchat" channel on irc.freenode.net and
see your bot join as well.
Now, as you probably guessed, you can't really /do/ much yet.
If you talk to the bot, it'll send messages to the terminal informing
you as such, but it's hardly a chat bot if it's not chatting yet.

So let's do the most boring (and annoying) thing possible.
Let's get it to echo whatever we say back to us.
Change handle-line to this:

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (<- irc-bot (actor-id irc-bot) 'send-line channel
        (format #f "Bawwwwk! ~a says: ~a" speaker line)))
#+END_SRC

This will do exactly what it looks like: repeat back whatever anyone
says like an obnoxious parrot.
Give it a try, but don't keep it running for too long... this
bot is so annoying it's likely to get banned from whatever channel
you put it in.

This method handler does have the advantage of being simple though.
It introduces a new concept simply... sending a message!
Whenever you see "<-", you can think of that as saying "send this
message".
The arguments to "<-" are as follows: the actor sending the message,
the id of the actor the message is being sent to, the "action" we
want to invoke (a symbol), and the rest are arguments to the
"action handler" which is in this case send-line (with itself takes
two arguments: the channel our bot should send a message to, and
the line we want it to spit out to the channel).

(Footnote: 8sync's name for sending a message, "<-", comes from older,
early lisp object oriented systems which were, as it turned out,
inspired by the actor model!
Eventually message passing was dropped in favor of something called
"generic functions" or "generic methods"
(you may observe we made use of such a thing in extending
handle-line).
Many lispers believe that there is no need for message passing
with generic methods and some advanced functional techniques,
but in a concurrent environment message passing becomes useful
again, especially when the communicating objects / actors are not
in the same address space.)

Normally in the actor model, we don't have direct references to
an actor, only an identifier.
This is for two reasons: to quasi-enforce the "shared nothing"
environment (actors absolutely control their own resources, and
"all you can do is send a message" to request that they modify
them) and because... well, you don't even know where that actor is!
Actors can be anything, and anywhere.
It's possible in 8sync to have an actor on a remote hive, which means
the actor could be on a remote process or even remote machine, and
in most cases message passing will look exactly the same.
(There are some exceptions; it's possible for two actors on the same
hive to "hand off" some special types of data that can't be serialized
across processes or the network, eg a socket or a closure, perhaps even
one with mutable state.
This must be done with care, and the actors should be careful both
to ensure that they are both local and that the actor handing things
off no longer accesses that value to preserve the actor model.
But this is an advanced topic, and we are getting ahead of ourselves.)
We have to supply the id of the receiving actor, and usually we'd have
only the identifier.
But since in this case, since the actor we're sending this to is
ourselves, we have to pass in our identifier, since the Hive won't
deliver to anything other than an address.

Astute readers may observe, since this is a case where we are just
referencing our own object, couldn't we just call "sending a line"
as a method of our own object without all the message passing?
Indeed, we do have such a method, so we /could/ rewrite handle-line
like so:

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (irc-bot-send-line irc-bot channel
                       (format #f "Bawwwwk! ~a says: ~a" speaker line)))
#+END_SRC

... but we want to get you comfortable and familiar with message
passing, and we'll be making use of this same message passing shortly
so that /other/ actors may participate in communicating with IRC
through our IRC bot.

Anyway, our current message handler is simply too annoying.
What would be much more interesting is if we could recognize
when an actor could repeat messages /only/ when someone is speaking
to it directly.
Luckily this is an easy adjustment to make.

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (define my-name (irc-bot-username irc-bot))
    (define (looks-like-me? str)
      (or (equal? str my-name)
          (equal? str (string-concatenate (list my-name ":")))))
    (when (looks-like-me?)
      (<- irc-bot (actor-id irc-bot) 'send-line channel
          (format #f "Bawwwwk! ~a says: ~a" speaker line))))
#+END_SRC

This is relatively straightforward, but it isn't very interesting.
What we would really like to do is have our bot respond to individual
"commands" like this:

#+BEGIN_SRC text
  <foo-user> examplebot: hi!
  <examplebot> Oh hi foo-user!
  <foo-user> examplebot: botsnack
  <examplebot> Yippie! *does a dance!*
  <foo-user> examplebot: echo I'm a very silly bot
  <examplebot> I'm a very silly bot
#+END_SRC

Whee, that looks like fun!
To implement it, we're going to pull out Guile's pattern matcher.

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (define my-name (irc-bot-username irc-bot))
    (define (looks-like-me? str)
      (or (equal? str my-name)
          (equal? str (string-concatenate (list my-name ":")))))
    (match (string-split line #\space)
      (((? looks-like-me? _) action action-args ...)
       (match action
         ;; The classic botsnack!
         ("botsnack"
          (<- irc-bot (actor-id irc-bot) 'send-line channel
              "Yippie! *does a dance!*"))
         ;; Return greeting
         ((or "hello" "hello!" "hello." "greetings" "greetings." "greetings!"
              "hei" "hei." "hei!" "hi" "hi!")
          (<- irc-bot (actor-id irc-bot) 'send-line channel
              (format #f "Oh hi ~a!" speaker)))
         ("echo"
          (<- irc-bot (actor-id irc-bot) 'send-line channel
              (string-join action-args " ")))

         ;; --->  Add yours here <---

         ;; Default
         (_
          (<- irc-bot (actor-id irc-bot) 'send-line channel
              "*stupid puppy look*"))))))
#+END_SRC

Parsing the pattern matcher syntax is left as an exercise for the
reader.

If you're getting the sense that we could make this a bit less wordy,
you're right:

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    (define my-name (irc-bot-username irc-bot))
    (define (looks-like-me? str)
      (or (equal? str my-name)
          (equal? str (string-concatenate (list my-name ":")))))
    (define (respond respond-line)
      (<- irc-bot (actor-id irc-bot) 'send-line channel
          respond-line))
    (match (string-split line #\space)
      (((? looks-like-me? _) action action-args ...)
       (match action
         ;; The classic botsnack!
         ("botsnack"
          (respond "Yippie! *does a dance!*"))
         ;; Return greeting
         ((or "hello" "hello!" "hello." "greetings" "greetings." "greetings!"
              "hei" "hei." "hei!" "hi" "hi." "hi!")
          (respond (format #f "Oh hi ~a!" speaker)))
         ("echo"
          (respond (string-join action-args " ")))

         ;; --->  Add yours here <---

         ;; Default
         (_
          (respond "*stupid puppy look*"))))))
#+END_SRC

Okay, that looks pretty good!
Now we have enough information to build an IRC bot that can do a lot
of things.
Take some time to experiment with extending the bot a bit before
moving on to the next section!
What cool commands can you add?

** An intermission: about live hacking

This section is optional, but highly recommended.
It requires that you're a user of GNU Emacs.
If you aren't, don't worry... you can forge ahead and come back in case
you ever do become an Emacs user.
(If you're more familiar with Vi/Vim style editing, I hear good things
about Spacemacs...)

So you may have noticed while updating the last section that the
start/stop cycle of hacking isn't really ideal.
You might either edit a file in your editor, then run it, or
type the whole program into the REPL, but then you'll have to spend
extra time copying it to a file.
Wouldn't it be nice if it were possible to both write code in a
file and try it as you go?
And wouldn't it be even better if you could live edit a program
while it's running?

Luckily, there's a great Emacs mode called Geiser which makes
editing and hacking and experimenting all happen in harmony.
And even better, 8sync is optimized for this experience.
8sync provides easy drop-in "cooperative REPL" support, and
most code can be simply redefined on the fly in 8sync through Geiser
and actors will immediately update their behavior, so you can test
and tweak things as you go.

Okay, enough talking.  Let's add it!
Redefine run-bot like so:

#+BEGIN_SRC scheme
  (define* (run-bot #:key (username "examplebot")
                    (server "irc.freenode.net")
                    (channels '("##botchat"))
                    (repl-path "/tmp/8sync-repl"))
    (define hive (make-hive))
    (define irc-bot
      (hive-create-actor* hive <my-irc-bot> "irc-bot"
                          #:username username
                          #:server server
                          #:channels channels))
    (define repl-manager
      (hive-create-actor* hive <repl-manager> "repl"
                          #:path repl-path))

    (run-hive hive (list (bootstrap-message hive irc-bot 'init)
                         (bootstrap-message hive repl-manager 'init))))
#+END_SRC

If we put a call to run-bot at the bottom of our file we can call it,
and the repl-manager will start something we can connect to automatically.
Horray!
Now when we run this it'll start up a REPL with a unix domain socket at
the repl-path.
We can connect to it in emacs like so:

: M-x geiser-connect-local <RET> guile <RET> /tmp/8sync-repl <RET>

Okay, so what does this get us?
Well, we can now live edit our program.
Let's change how our bot behaves a bit.
Let's change handle-line and tweak how the bot responds to a botsnack.
Change this part:

#+BEGIN_SRC scheme
  ;; From this:
  ("botsnack"
   (respond "Yippie! *does a dance!*"))

  ;; To this:
  ("botsnack"
   (respond "Yippie! *catches botsnack in midair!*"))
#+END_SRC

Okay, now let's evaluate the change of the definition.
You can hit "C-M-x" anywhere in the definition to re-evaluate.
(You can also position your cursor at the end of the definition and press
"C-x C-e", but I've come to like "C-M-x" better because I can evaluate as soon
as I'm done writing.)
Now, on IRC, ask your bot for a botsnack.
The bot should give the new message... with no need to stop and start the
program!

Let's fix a bug live.
Our current program works great if you talk to your bot in the same
IRC channel, but what if you try to talk to them over private message?

#+BEGIN_SRC text
IRC> /query examplebot
<foo-user> examplebot: hi!
#+END_SRC

Hm, we aren't seeing any response on IRC!
Huh?  What's going on?
It's time to do some debugging.
There are plenty of debugging tools in Guile, but sometimes the simplest
is the nicest, and the simplest debugging route around is good old
fashioned print debugging.

It turns out Guile has an under-advertised feature which makes print
debugging really easy called "pk", pronounced "peek".
What pk accepts a list of arguments, prints out the whole thing,
but returns the last argument.
This makes wrapping bits of our code pretty easy to see what's
going on.
So let's peek into our program with pk.
Edit the respond section to see what channel it's really sending
things to:

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    ;; [... snip ...]
    (define (respond respond-line)
      (<- irc-bot (actor-id irc-bot) 'send-line (pk 'channel channel)
          respond-line))
    ;; [... snip ...]
    )
#+END_SRC

Re-evaluate.
Now let's ping our bot in both the channel and over PM.

#+BEGIN_SRC text
;;; (channel "##botchat")

;;; (channel "sinkbot")
#+END_SRC

Oh okay, this makes sense.
When we're talking in a normal multi-user channel, the channel we see
the message coming from is the same one we send to.
But over PM, the channel is a username, and in this case the username
we're sending our line of text to is ourselves.
That isn't what we want.
Let's edit our code so that if we see that the channel we're sending
to looks like our own username that we respond back to the sender.
(We can remove the pk now that we know what's going on.)

#+BEGIN_SRC scheme
  (define-method (handle-line (irc-bot <my-irc-bot>) speaker channel
                              line emote?)
    ;; [... snip ...]
    (define (respond respond-line)
      (<- irc-bot (actor-id irc-bot) 'send-line
          (if (looks-like-me? channel)
              speaker    ; PM session
              channel)   ; normal IRC channel
          respond-line))
    ;; [... snip ...]
    )
#+END_SRC

Re-evaluate and test.

#+BEGIN_SRC text
IRC> /query examplebot
<foo-user> examplebot: hi!
<examplebot> Oh hi foo-user!
#+END_SRC

Horray!

** Battle bot!

** Adding a "rankings" web page

** Writing our own <irc-bot> from scratch

* API reference