Tuesday, March 31, 2015

Getting bitcoin-ruby-blockchain and postgres working

These are just rough notes for myself. Perhaps you'll get something out of them too. At the moment I appear to be trying random things until I get it to work, so many of the steps below may not be necessary.

This post assumes you've set up a bitcoin ruby Ubuntu machine as per the previous posts. The following instructions should get you up and running with an (almost) complete postgres database using mhannes bitcoin-ruby-blockchain gem.

$ sudo git clone https://github.com/mhanne/bitcoin-ruby-blockchain.git
$ cd bitcoin-ruby-blockchain/
$ sudo gem build bitcoin-ruby-blockchain.gemspec
$ sudo gem install bitcoin-ruby-blockchain-0.0.1.gem

That gets the bitcoin ruby blockchain gem installed. Now you need to download the postgres database dump from webbtc.com. It's about 38Gb at the time of writing, so it will take about 10 hours. Go to http://dumps.webbtc.com/bitcoin/ and get the sql.gz file.

I couldn't unzip it with the default archive manager in Ubuntu, and had to resort to using 7zip on Windows. The resulting file is an 85Gb text file full of SQL commands. It's probably best to store it on an external drive as you won't be needing it more than once, hopefully. Again, unzipping took about 3 hours. Correction: use gunzip from a terminal, like so, and it takes less than two hours:

$ gunzip bitcoin_2015-03-02.sql.gz 

Now you need to get postgres running. Unlike Sqlite3, which stores the whole database in one file, postgres is a more professional (and hence more complicated) database system. It stores everything in a folder structure, and a server is run for database applications to connect to.

Run the following:

$ sudo apt-get update
$ sudo apt-get install postgresql postgresql-contrib
sudo apt-get install libpq-dev
$ sudo gem install pg
$ sudo apt-get update
$ sudo 

This ins talls all the backend postgres stuff. You'll need an administration tool. Get it from the Ubuntu software center; it's called pgAdmin III.

Open bitcoin-ruby/examples/balance.rb in a text editor, and change line 13 to read:

store = Bitcoin::Storage.sequel(:db => "sqlite://bitcoin.db")

At this point you can check you've got everything except an actual database by running 

$ sudo ruby balance.rb 1Q2TWHE3GMdB6BZKafqwxXtWAWgFt5Jvm3

If everything installed correctly you'll get an error that you can't connect to a database server.

Create a new postgres user

$ sudo createuser postgres
Select postgres as password, and leave all other info empty
$ sudo adduser postgres sudo
$ sudo su - postgres

But the server wasn't started. I found this page: http://www.rogerpence.com/2015/01/02/install-postgres-9-4-on-ubuntu/ and section 3 almost worked. Correcting the triple line to what's below worked, and a message claimed the server was started.

wget --quiet -O - https://apt.postgresql.org/pub/repos/apt/ACCC4CF8.asc | sudo apt-key add -

Start pgAdmin III. Click on the power plug symbol at the top left. Enter a name of bitcoin, a location of localhost and make sure you enter the postgres password.

If it doesn't work, try resetting the postgres password in psql using http://stackoverflow.com/questions/19774742/pgadmin3-server-connection-and-authentication-error

Expand the tree till you see Databases (1) and right click, then select add new database and create another - just give it the name bitcoin, everything else is okay.

Add a login role similarly and create a superuser called webbtc. Give it all the rights all the tabs allow you, and don't set an expiry.

Run pdump to execute the SQL commands in the downloaded unzipped file on the new database:

$ psql -d bitcoin -f bitcoin.sql

After about eight hours I had a working Postgres database with the blockchain up to 1 March 2015 running on my Ubuntu laptop. The bitcoin.rb balance program needed editing to use Postgres rather than mySQL, but after that it was possible to query the database for address balances. 

Setting up a litecoin miner on an Ubuntu laptop with a Radeon GPU

These are just rough notes for myself. Perhaps you'll get something useful out of them too.

Download cgminer from http://downloadandroidrom.com/file/zedomax/linux/cgminer/cgminer-3.7.2-x86_64-built.tar.bz2 but this could be bad. Most links are now dead. We need 3.7.2 because it's the last cgminer build with scrypt support and GPU mining support.

wget http://www2.ati.com/drivers/beta/amd-catalyst-14.1-betav1.3-linux-x86.x86_64.zip --referer="http://support.amd.com/en-us/download/desktop?os=Linux+x86"
unzip amd-catalyst-14.1-betav1.3-linux-x86.x86_64.zip
sudo sh *.run --buildpkg Ubuntu/raring
sudo dpkg -i fglrx*.deb

So let's build it from git.

$ mkdir litecoin
$ cd litecoin
$ git clone https://github.com/ckolivas/cgminer.git
$ cd cgminer
$ git checkout tags/v3.7.2

$ sudo apt-get install build-essential autoconf automake libtool pkg-config libcurl3-dev libudev-dev
$ sudo apt-get install libcurl4-openssl-dev
$ sudo ./autogen.sh m4_pattern_allow

$ autoconf
$ automake

$ sudo apt-get install libcurl4-openssl-dev libncurses5-dev pkg-config libtool yasm autogen automake

AMD Radeon Catalyst http://www2.ati.com/drivers/linux/amd-catalyst-14-9-linux-x86-x86-64.zip
ADL SDK http://developer.amd.com/tools-and-sdks/graphics-development/display-library-adl-sdk/#

$ unzip amd-catalyst-14-9-linux-x86-x86-64.zip
$ cd fglrx-14.301.1001
$ sudo chmod +x amd-driver-installer*.run
$ sudo apt-get install dh-modaliases execstack debhelper dkms lib32gcc1 
$ sudo ./amd-driver-installer*.run

$ sudo aticonfig --initial -f --adapter=all



Sunday, October 5, 2014

A simple second Ruby bitcoin program

In order to get this to run you need install the ffi gem with:

     $ sudo gem install ffi

Now save the following code in a file (I called mine check.rb):

     require 'bitcoin'
   
     privwif = ARGV[0]
   
     mykey = Bitcoin::Key.from_base58(privwif)
     puts mykey.addr

If you run this with a valide base58 private key, it will output the corresponding public key. But there is one problem - if the private key is invalid, you get a complicated error. So let's add a section to check for this. Replace your file with the following:

     require 'bitcoin'
   
     privwif = ARGV[0]
   
     begin
       mykey = Bitcoin::Key.from_base58(privwif)
       puts mykey.addr
     rescue
         puts "Invalid private key."
     end

As soon as the wrong key error occurs the program jumps to the rescue section and prints out that there is an invalid key. However, if any other error occurs it will also print that message, so the program needs extending.

Sunday, September 21, 2014

Writing your first bitcoin-ruby program

Imagine you wrote down a WIF private bitcoin key, but because your handwriting was unclear or you wrote it in a hurry, it turns out one character was copied down wrong.

Here is the incorrect WIF: 5KL6HJfzZ4Uj4CJRFYT11z2W1t5kbULaDqHd7rn6ktLwT6sgyrY

You want to go through each character in the WIF string, replacing each single character in turn with one of the 58 valid characters for the WIF format, and check if it's a valid WIF key.

#!/usr/bin/env ruby
# First the required includes for bitcoin and openssl
require 'bitcoin'
require 'openssl'

# This function performs the correct checksum on a proposed WIF key
# to determine if it is a genuine WIF key
# See http://gobittest.appspot.com/PrivateKey (TP's Go Bitcoin Tests) for an explanation
def wif_valid?(hex)
hex = Bitcoin::base58_to_hex(hex)
hex1 = hex[0..-9]
hex1 = Bitcoin::checksum(hex1)
hex2 = hex.reverse[0..7].reverse
if hex1 == hex2
return true
end
return false
end

# This is our one character wrong WIF
badwif = "5KL6HJfzZ4Uj4CJRFYT11z2W1t5kbULaDqHd7rn6ktLwT6sgyrY"

# And this function takes a WIF, and loops through each character,
# changing it by one, and checking if it is now valid.
# It prints out any valid WIF it finds (there should only be one)
def one_alter(awif)
code_string = "123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz"
newwif = ""
for i in 1..awif.length-1
for j in 0..code_string.length-1
newwif = awif
   # Next line is a  hack to ensure that newwif is a new string 
   # and not a pointer to the same string as awif
newwif = newwif + ""
newwif[i] = code_string[j]
if wif_valid?(newwif) == true
    # print out the newwif if it is a correct key
puts newwif
end
end
end
end

one_alter badwif

So lets see what happens when we run it:

bitcoin-ruby$ ruby programs/address.rb 

5KL6HJfzZ4Uj4CJRFYT11z2W1t5kbULaDqHd7rn6ktLwT5sgyrY

There you go - the one valid WIF address within one character of the bad one is retrieved. And it's corresponding public address is 1MUDdayQ7Af39bT9mA6zRYd7ehEeuqFzLZ . Note: no balance, and unlike to have one for long from now on, given that I've committed the cardinal sin of bitcoin, and published the private key.

Setting up a Ruby Bitcoin Machine, part 2

Now we are going to install Ruby 1.9.3 and bitcoin-ruby plus required libraries.

1. First Ruby: go to the Ubuntu Software Center, and type 'ruby' in the search box. Select and install the standard 1.9.3 Ruby package. This installs the basic Ruby environment.

2. You're going to need a few extras to get bitcoin-ruby to run.

$ sudo apt-get install ruby1.9.1-dev
$ sudo apt-get install build-essential g++
$ sudo gem install eventmachine
$ sudo apt-get install openssl
$ sudo apt-get install sqlite libsqlite3-dev
$ sudo gem install sqlite3
$ sudo gem install sequel
$ sudo gem install em-dns
$ sudo apt-get install git 

3. And now bitcoin-ruby itself:

$ sudo git clone https://github.com/lian/bitcoin-ruby.git; cd bitcoin-ruby
$ ruby bin/bitcoin_node

The last command should execute without any errors, which shows that the system is working. Stop it with Ctrl+C.

The final step is to load all the blockchain data into an sqlite3 database, with the following command:

$ ruby bin/bitcoin_node --storage sequel::sqlite://bitcoin.db --import ~/.bitcoin/blocks --skip-validation

This reads the bitcoin blockchain from your bitcoin client directory, and parses it before loading relevant information into a bitcoin.db database in your bitcoin-ruby directory.

If you get an error about opening the database like SQLite3::CantOpenException: unable to open database file (Sequel::DatabaseConnectionError) try running the previous command with sudo in front.

Once this has been running for a bit you can check, for example, that the balance.rb program works:

$ ruby examples/balance.rb 12c6DSiU4Rq3P4ZxziKxzrL5LmMBrzjrJX

This should return:

INFO  storage: opened sequel store sqlite://bitcoin.db
Balance: 50.00000000

It's the first address used in the first (genesis) block of the blockchain. As these coins can never be redeemed it will always have a non-zero balance.

Now that we've shown that it all works, it's time to install the Ruby gem for bitcoin-ruby so the various methods and objects can be called from anywhere:

$ sudo gem build bitcoin-ruby.gemspec && sudo gem install bitcoin-ruby-0.0.5.gem

Right, with that all working, you can start writing Ruby programs to do stuff with bitcoin.

Setting up a Ruby Bitcoin Machine, part 1

The next few posts are a record for myself, on how to install and set up a Linux machine to allow me to write programs that do bitcoin related things in Ruby, using the bitcoin-ruby system.

Step 1: Install Ubuntu 14.04 on a spare laptop. Make sure it's a 64 bit AMD machine, and that you install the right version. The required file can be obtained from http://www.ubuntu.com/download/desktop

It's free if you select the "No, take me to the download" link at the bottom right. An iso file called ubuntu-14.04.1-desktop-amd64.iso will appear in your download folder. Burn it to a DVD as an image.

Reboot the machine you want to install on, and boot from the DVD. Choose to install it, and don't select home folder encryption because it will slow down your programs.

Step 2: Boot up and get all the updates for Ubuntu (in the top left swirl icon, type updater and click on "Software Updater"). Then download the bootstrap.dat bitcoin blockchain by selecting the torrent bootstrap.dat.torrent from the page http://sourceforge.net/projects/bitcoin/files/Bitcoin/blockchain/

This is a multi-gigabyte file so the torrent downloaded will take a while.

In the meantime install the Bitcoin-qt client. Open a terminal (Ctrl+Alt+T) and type:

$ sudo apt-add-repository ppa:bitcoin/bitcoin
$ sudo apt-get update
$ sudo apt-get install bitcoin-qt

Click on the swirl icon and type bitcoin. The Bitcoin program with the orange logo should appear. Run it, but stop it after about 5 minutes.

When bootstrap.dat has downloaded copy it from the Downloads folder to the .bitcoin folder (you need to select Show Hidden Files from the View menu in the file explorer, or  user Ctrl+H).

Now you can run the Bitcoin client, and it will load the blocks off disk (this took about 4 hours for my laptop), and will then catch up with the remaining blocks from the network (another 16 hours or so). Now there is a complete copy of the blockchain in ~/.bitcoin/blocks/





Saturday, March 22, 2014

Bitcoin theft

A friend of mine had the misfortune of storing his bitcoins in Flexcoin, the Canadian Bitcoin Bank that was robbed of 800 bitcoins by a hacker earlier this week. Although of course, when I write "Canadian Bitcoin Bank", what it actually turns out to be is a company registered to a rented mailbox in the city of Edmonton Alberta, according to provincial records, with its sole director and voting shareholder one James Andrew Gray, who also gave the rented mailbox as his address. This according to Reuters.

The case is similar to Trade Fortress, another robbed Bitcoin Bank, that time in Australia, which lost 4100 bitcoins, allegedly to a hacker. Again, the "Bank" turned out to be a young Australian developer (I couldn't find out what his name was though).
It's tricky though to determine from a web page whether you're dealing with a teenager working from his parents' spare bedroom, or a large established company with proper offices in the bitcoin world.
However, even larger companies with proper offices don't offer any guarantees. Mt. Gox was supposedly one of those proper companies, but managed to somehow lose over 650,000 bitcoins. Again, information has since leaked out that the company didn't use bug tracking systems or source control for the code that it's developers produced, and the CEO was more interested in setting up a cafe that accepted bitcoins than running the exchange.
The only truly safe way of storing your bitcoins seems to be:

  1. Get yourself a computer that's never been on the internet
  2. Copy OpenSSL across to it using a USB stick
  3. Generate a private key by tossing a coin 256 times, carefully writing down the result
  4. Spend half a day checking you've typed the private key into your disconnected laptop properly, and generate a public key
  5. Send your bitcoins to the address associated with the public key

And even that's not guaranteed.

This is, in my opinion, the biggest problem bitcoin faces.