Quick summary

Public and private keys come in pairs. What is encrypted by one, can be decrypted by the other.

When used for server access (i.e. through SSH or Putty), this makes encrypting passwords in transit superfluous. The user keeps the private key, server gets the public version.

More info on WikiPedia.