Commit b697b5b1 authored by Artem B's avatar Artem B

merge latest electrumx changes

parents 86b058ef 302cfcfe
......@@ -22,7 +22,7 @@ There is also an `Dockerfile`_ available .
.. _installer:
.. _Dockerfile:
.. _Dockerfile:
......@@ -103,16 +103,28 @@ Roadmap
Version 1.2
IMPORTANT: this release changes script hash indexing in the database,
IMPORTANT: version 1.2 changes script hash indexing in the database,
so you will need to rebuild your databases from scratch. Running this
version will refuse to open the DB and not corrupt it, so you can
revert to 1.1.x if you wish. The initial synchronisation process
should be around 10-15% faster than 1.1, owing to this change and
Justin Arthur's optimisations from 1.1.1.
Version 1.2.1
- remove IRC support. Most coins had empty IRC channels. Those that
don't have peers populated.
- use estimatesmartfee RPC call if available (SomberNight)
- new/updated coins: Emercoin (Sergii Vakula), Bitcoin Gold (erasmospunk),
Monacoin testnet (Wakiyama P), sibcoin (53r63rn4r), Komodo and Monaize
(cipig), Hush (Duke Leto)
- doc updates (fr3aker)
- issues fixed: `#302`_
Version 1.2
- separate P2PKH from P2PK entries in the history and UTXO databases.
These were previously amalgamated by address as that is what
electrum-server used to do. However Electrum didn't handle P2PK
......@@ -218,54 +230,18 @@ Version 1.0.15
- add Vertcoin support (erasmospunk)
- update Faircoin (thokon00)
Version 1.0.14
- revert the changes to mempool handling of 1.0.13; I think they introduced
a notification bug
Version 1.0.13
- improve mempool handling and height notifications
- add bitcoin-segwit as a new COIN
Version 1.0.12
- handle legacy daemons, add support for Blackcoin and Peercoin (erasmospunk)
- implement history compression; can currently only be done from a script
with the server down
- Add dockerfile reference (followtheart)
- doc, runfile fixes (Henry, emilrus)
- add bip32 implementation, currently unused
- daemon compatibility improvements (erasmospunk)
- permit underscores in hostnames, updated Bitcoin server list
Version 1.0.11
- disable IRC for bitcoin mainnet
- remove dead code, allow custom Daemon & BlockProcessor classes (erasmospunk)
- add SERVER_(SUB)VERSION to banner metavariables (LaoDC)
- masternode methods for Dash (TheLazier)
- allow multiple P2SH address versions, implement for Litecoin (pooler)
- update Bitcoin's TX_COUNT and block height (JWU42)
- update BU nolnet parameters
- fix diagnostic typo (anduck)
- Issues fixed: `#180`_
**Neil Booth**
.. _#180:
.. _#223:
.. _#251:
.. _#277:
.. _#287:
.. _#302:
.. _docs/HOWTO.rst:
.. _docs/ENVIRONMENT.rst:
.. _docs/PROTOCOL.rst:
......@@ -3,6 +3,9 @@
# install electrumx
# Remove "raspi-copies-and-fills" as it breaks the upgrade process
sudo apt-get purge raspi-copies-and-fills
# upgrade raspbian to 'stretch' distribution
sudo echo 'deb testing main contrib non-free rpi' > /etc/apt/sources.list.d/stretch.list
sudo apt-get update
......@@ -17,7 +20,6 @@ sudo apt install libreadline6-dev/stable libreadline6/stable
sudo apt-get install libleveldb-dev
sudo apt-get install git
sudo pip3 install plyvel
sudo pip3 install irc
# install electrumx
git clone
......@@ -20,8 +20,6 @@ export TCP_PORT=50001
export SSL_PORT=50002
# visibility
export IRC=
export IRC_NICK=hostname
export RPC_PORT=8000
......@@ -81,8 +81,3 @@ Database
The underlying data store, made up of the DB backend (such as
`leveldb`) and the host filesystem.
Handles advertising of ElectrumX services via IRC.
......@@ -92,12 +92,18 @@ These environment variables are optional:
If set ElectrumX will serve TCP clients on **HOST**:**TCP_PORT**.
**Note**: ElectrumX will not serve TCP connections until it has
fully caught up with your daemon.
* **SSL_PORT**
If set ElectrumX will serve SSL clients on **HOST**:**SSL_PORT**.
If set then SSL_CERTFILE and SSL_KEYFILE must be defined and be
filesystem paths to those SSL files.
**Note**: ElectrumX will not serve SSL connections until it has
fully caught up with your daemon.
* **RPC_HOST**
The host or IP address that the RPC server will listen on and
......@@ -143,6 +149,10 @@ These environment variables are optional:
+ **$DONATION_ADDRESS** is replaced with the address from the
**DONATION_ADDRESS** environment variable.
See for a script
that updates a banner file periodically with useful statistics about
fees, last block time and height, etc.
As for **BANNER_FILE** (which is also the default) but shown to
......@@ -296,9 +306,7 @@ some of this.
By default peer discovery happens over the clear internet. Set this
to non-empty to force peer discovery to be done via the proxy. This
might be useful if you are running a Tor service exclusively and
wish to keep your IP address private. **NOTE**: in such a case you
should leave **IRC** unset as IRC connections are *always* over the
normal internet.
wish to keep your IP address private.
......@@ -317,8 +325,8 @@ some of this.
Server Advertising
These environment variables affect how your server is advertised, both
by peer discovery (if enabled) and IRC (if enabled).
These environment variables affect how your server is advertised
by peer discovery (if enabled).
......@@ -357,27 +365,6 @@ by peer discovery (if enabled) and IRC (if enabled).
Use the following environment variables if you want to advertise
connectivity on IRC:
* **IRC**
Set to anything non-empty to advertise on IRC. ElectrumX connects
to IRC over the clear internet, always.
* **IRC_NICK**
The nick to use when connecting to IRC. The default is a hash of
**REPORT_HOST**. Either way a prefix will be prepended depending on
**COIN** and **NET**.
If **REPORT_HOST_TOR** is set, an additional connection to IRC
happens with '_tor' appended to **IRC_NICK**.
......@@ -18,10 +18,6 @@ Python3 ElectrumX uses asyncio. Python version >= 3.6 is
DB Engine I use `plyvel`_ 0.9, a Python interface to LevelDB.
A database engine package is required but others
are supported (see **Database Engine** below).
`IRC`_ Python IRC package. Only required if you enable
IRC; ElectrumX will happily serve clients that
try to connect directly. I use 15.0.4 but
older versions likely are fine.
`x11_hash`_ Only required for DASH. Python X11 Hash package. Only
required if for Dash. Version 1.4 tested.
================ ========================
......@@ -260,6 +256,10 @@ machine. **DB_CACHE** set to 1,800. LevelDB.
For chains other than bitcoin-mainnet sychronization should be much
**Note**: ElectrumX will not serve normal client connections until it
has fully synchronized and caught up with your daemon. However
LocalRPC connections are served at all times.
Terminating ElectrumX
......@@ -368,10 +368,7 @@ your sign request to identify your server. They are not currently
checked by the client except for the validity date. When asked for a
challenge password just leave it empty and press enter::
$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
$ openssl rsa -passin pass:x -in server.pass.key -out server.key
writing RSA key
$ rm server.pass.key
$ openssl genrsa -out server.key 2048
$ openssl req -new -key server.key -out server.csr
Country Name (2 letter code) [AU]:US
......@@ -414,7 +411,6 @@ You can then set the port as follows and advertise the service externally on the
.. _`runit`:
.. _`aiohttp`:
.. _`pylru`:
.. _`IRC`:
.. _`x11_hash`:
.. _`contrib/python3.6/`:
.. _`contrib/raspberrypi3/`:
Peer Discovery
This is a suggestion of a peer discovery prtocol as a way to gradually
move off depending on IRC.
It will be implemented in ElectrumX from version 0.11.0
This was imlpemented in ElectrumX as of version 0.11.0. Support for
IRC peer discovery was removed in ElectrumX version 1.2.1.
Peer Database
......@@ -154,14 +151,12 @@ Unknown keys should be silently ignored.
* **protocol_min**
Strings that are the minimum and maximum Electrum protocol versions
this server speaks. The maximum value should be the same as what
would suffix the letter **v** in the IRC real name. Example: "1.1".
this server speaks. Example: "1.1".
* **pruning**
An integer, the pruning limit. Omit or set to *null* if there is no
pruning limit. Should be the same as what would suffix the letter
**p** in the IRC real name.
pruning limit.
server.add_peer RPC call
......@@ -184,18 +179,6 @@ calls to this method from a single connection.
The result should be True if accepted and False otherwise.
Other server implementations may not have implemented the peer
discovery protocol yet. Whilst we transition away from IRC, in order
to keep these servers in the connected peer set, having one or two in
the hard-coded peer list used to seed this process should suffice.
Any peer on IRC will report other peers on IRC, and so if any one of
them is known to any single peer implementing this protocol, they will
all become known to all peers quite rapidly.
Notes to Implementators
......@@ -75,6 +75,58 @@ from and including the server's response to this call will use the
negotiated protocol version.
Script Hashes
A script hash is the hash of the binary bytes of the locking script
(ScriptPubKey), expressed as a hexadecimal string. The hash function
to use is given by the "hash_function" member of `server.features`
(currently "sha256" only). Like for block and transaction hashes, when
converting the big-endian binary hash to a hexadecimal string the
least-significant byte appears first, and the most-significant byte
For example, the legacy Bitcoin address from the genesis block
has P2PKH script
with SHA256 hash
which is sent to the server reversed as
By subscribing to this hash you can find P2PKH payments to that address.
One public key for that address (the genesis block public key) is
which has P2PK script
with SHA256 hash
which is sent to the server reversed as
By subscribing to this hash you can find P2PK payments to that public
key. Note the Genesis block coinbase is unspendable and therefore not
Protocol Version 1.0
......@@ -846,8 +898,7 @@ Get a list of features and services supported by the server.
* **protocol_min**
Strings that are the minimum and maximum Electrum protocol versions
this server speaks. The maximum value should be the same as what
would suffix the letter **v** in the IRC real name. Example: "1.1".
this server speaks. Example: "1.1".
* **pruning**
......@@ -869,5 +920,35 @@ Get a list of features and services supported by the server.
"hash_function": "sha256"
Protocol Version 1.2
Protocol version 1.2 is the same as version `1.1` except for the
addition of a new method `mempool.get_fee_histogram`.
All methods with taking addresses are deprecated, and will be removed
at some point in the future. You should update your code to use
`Script Hashes`_ and the scripthash methods introduced in protocol 1.1
Request a histogram of the fee rates paid by transactions in the memory
pool, weighted by transaction size.
The histogram is an array of [fee, vsize] pairs, where vsize_n is
the cumulative virtual size of mempool transactions with a fee rate
in the interval [fee_(n-1), fee_n)], and fee_(n-1) > fee_n.
Fee intervals may have variable size. The choice of appropriate
intervals is currently not part of the protocol.
.. _JSON RPC 1.0:
.. _JSON RPC 2.0:
......@@ -38,7 +38,7 @@ The following commands are available:
"groups": 2, # The number of session groups
"logged": 0, # The number of sessions being logged
"paused": 0, # The number of paused sessions.
"peers": 62, # Number of peer servers (from IRC)
"peers": 62, # Number of peer servers
"pid": 126275, # The server's process ID
"requests": 0, # Number of unprocessed requests
"sessions": 85, # Number of current sessions (connections)
......@@ -153,14 +153,14 @@ The following commands are available:
Returns a list of peer electrum servers. This command takes no arguments.
Currently peer data is obtained via a peer discovery protocol; it
used to be taken from IRC.
Peer data is obtained via a peer discovery protocol documented in
* **add_peer**
Add a peer to the peers list. ElectrumX will schdule an immediate
connection attempt. This command takes a single argument: the
peer's "real name" as it would advertise itself on IRC.
peer's "real name" as it used to advertise itself on IRC.
.. code::
......@@ -186,3 +186,5 @@ The following commands are available:
Force a block chain reorg. This command takes an optional
argument - the number of blocks to reorg - which defaults to 3.
.. _docs/PEER_DISCOVERY.rst:
This diff is collapsed.
......@@ -71,7 +71,8 @@ class TxOutput(namedtuple("TxOutput", "value pk_script")):
class Deserializer(object):
'''Deserializes blocks into transactions.
External entry points are read_tx() and read_block().
External entry points are read_tx(), read_tx_and_hash(),
read_tx_and_vsize() and read_block().
This code is performance sensitive as it is executed 100s of
millions of times during sync.
......@@ -84,25 +85,32 @@ class Deserializer(object):
self.cursor = start
def read_tx(self):
'''Return a (Deserialized TX, TX_HASH) pair.
The hash needs to be reversed for human display; for efficiency
we process it in the natural serialized order.
start = self.cursor
'''Return a deserialized transaction.'''
return Tx(
self._read_le_int32(), # version
self._read_inputs(), # inputs
self._read_outputs(), # outputs
self._read_le_uint32() # locktime
), double_sha256(self.binary[start:self.cursor])
def read_tx_and_hash(self):
'''Return a (deserialized TX, tx_hash) pair.
The hash needs to be reversed for human display; for efficiency
we process it in the natural serialized order.
start = self.cursor
return self.read_tx(), double_sha256(self.binary[start:self.cursor])
def read_tx_and_vsize(self):
'''Return a (deserialized TX, vsize) pair.'''
return self.read_tx(), self.binary_length
def read_tx_block(self):
'''Returns a list of (deserialized_tx, tx_hash) pairs.'''
read_tx = self.read_tx
txs = [read_tx() for _ in range(self._read_varint())]
read = self.read_tx_and_hash
# Some coins have excess data beyond the end of the transactions
return txs
return [read() for _ in range(self._read_varint())]
def _read_inputs(self):
read_input = self._read_input
......@@ -198,18 +206,16 @@ class DeserializerSegWit(Deserializer):
read_varbytes = self._read_varbytes
return [read_varbytes() for i in range(self._read_varint())]
def read_tx(self):
'''Return a (Deserialized TX, TX_HASH) pair.
The hash needs to be reversed for human display; for efficiency
we process it in the natural serialized order.
def _read_tx_parts(self):
'''Return a (deserialized TX, tx_hash, vsize) tuple.'''
start = self.cursor
marker = self.binary[self.cursor + 4]
if marker:
return super().read_tx()
tx = super().read_tx()
tx_hash = double_sha256(self.binary[start:self.cursor])
return tx, tx_hash, self.binary_length
# Ugh, this is nasty.
start = self.cursor
version = self._read_le_int32()
orig_ser = self.binary[start:self.cursor]
......@@ -221,14 +227,27 @@ class DeserializerSegWit(Deserializer):
outputs = self._read_outputs()
orig_ser += self.binary[start:self.cursor]
base_size = self.cursor - start
witness = self._read_witness(len(inputs))
start = self.cursor
locktime = self._read_le_uint32()
orig_ser += self.binary[start:self.cursor]
vsize = (3 * base_size + self.binary_length) // 4
return TxSegWit(version, marker, flag, inputs, outputs, witness,
locktime), double_sha256(orig_ser), vsize
def read_tx(self):
return self._read_tx_parts()[0]
def read_tx_and_hash(self):
tx, tx_hash, vsize = self._read_tx_parts()
return tx, tx_hash
return TxSegWit(version, marker, flag, inputs,
outputs, witness, locktime), double_sha256(orig_ser)
def read_tx_and_vsize(self):
tx, tx_hash, vsize = self._read_tx_parts()
return tx, vsize
class DeserializerAuxPow(Deserializer):
......@@ -289,7 +308,6 @@ class TxJoinSplit(namedtuple("Tx", "version inputs outputs locktime")):
class DeserializerZcash(DeserializerEquihash):
def read_tx(self):
start = self.cursor
base_tx = TxJoinSplit(
self._read_le_int32(), # version
self._read_inputs(), # inputs
......@@ -302,7 +320,7 @@ class DeserializerZcash(DeserializerEquihash):
self.cursor += joinsplit_size * 1802 # JSDescription
self.cursor += 32 # joinSplitPubKey
self.cursor += 64 # joinSplitSig
return base_tx, double_sha256(self.binary[start:self.cursor])
return base_tx
class TxTime(namedtuple("Tx", "version time inputs outputs locktime")):
......@@ -315,21 +333,17 @@ class TxTime(namedtuple("Tx", "version time inputs outputs locktime")):
class DeserializerTxTime(Deserializer):
def read_tx(self):
start = self.cursor
return TxTime(
self._read_le_int32(), # version
self._read_le_uint32(), # time
self._read_inputs(), # inputs
self._read_outputs(), # outputs
self._read_le_uint32(), # locktime
), double_sha256(self.binary[start:self.cursor])
class DeserializerReddcoin(Deserializer):
def read_tx(self):
start = self.cursor
version = self._read_le_int32()
inputs = self._read_inputs()
outputs = self._read_outputs()
......@@ -339,13 +353,7 @@ class DeserializerReddcoin(Deserializer):
time = 0
return TxTime(
), double_sha256(self.binary[start:self.cursor])
return TxTime(version, time, inputs, outputs, locktime)
class DeserializerTxTimeAuxPow(DeserializerTxTime):
......@@ -253,6 +253,8 @@ class BlockProcessor(server.db.DB):'processed {:,d} block{} in {:.1f}s'
.format(len(blocks), s,
time.time() - start))
elif hprevs[0] != chain[0]:
await self.reorg_chain()
......@@ -511,7 +513,6 @@ class BlockProcessor(server.db.DB):
if self.caught_up_event.is_set():
if time.time() > self.next_cache_check:
self.next_cache_check = time.time() + 30
......@@ -125,9 +125,8 @@ class Controller(ServerBase):
return self.mempool.value(hashX)
def sent_tx(self, tx_hash):
'''Call when a TX is sent. Tells mempool to prioritize it.'''
'''Call when a TX is sent.'''
self.txs_sent += 1
def setup_bands(self):
bands = []
......@@ -206,15 +205,15 @@ class Controller(ServerBase):
self.next_log_sessions = time.time() + self.env.log_sessions
async def wait_for_bp_catchup(self):
'''Wait for the block processor to catch up, then kick off server
background processes.'''
'''Wait for the block processor to catch up, and for the mempool to
synchronize, then kick off server background processes.'''
await self.bp.caught_up_event.wait()'block processor has caught up')
await self.mempool.synchronized_event.wait()
def close_servers(self, kinds):
'''Close the servers of the given kinds (TCP etc.).'''
......@@ -272,27 +271,26 @@ class Controller(ServerBase):
sslc.load_cert_chain(env.ssl_certfile, keyfile=env.ssl_keyfile)
await self.start_server('SSL', host, env.ssl_port, ssl=sslc)
async def notify(self):
def notify_sessions(self, touched):
'''Notify sessions about height changes and touched addresses.'''
while True:
await self.mempool.touched_event.wait()
touched = self.mempool.touched.copy()
# Invalidate caches
hc = self.history_cache
for hashX in set(hc).intersection(touched):
del hc[hashX]
if self.bp.db_height != self.cache_height:
self.cache_height = self.bp.db_height
# Make a copy; self.sessions can change whilst await-ing
sessions = [s for s in self.sessions
if isinstance(s, self.coin.SESSIONCLS)]
for session in sessions:
await session.notify(self.bp.db_height, touched)
# Invalidate caches
hc = self.history_cache
for hashX in set(hc).intersection(touched):
del hc[hashX]
height = self.bp.db_height
if height != self.cache_height:
self.cache_height = height
# Height notifications are synchronous. Those sessions with
# touched addresses are scheduled for asynchronous completion
for session in self.sessions:
if isinstance(session, LocalRPC):
session_touched = session.notify(height, touched)
if session_touched is not None:
def notify_peers(self, updates):
'''Notify of peer updates.'''
......@@ -827,6 +825,14 @@ class Controller(ServerBase):
number = self.non_negative_integer(number)
return await self.daemon_request('estimatefee', [number])
def mempool_get_fee_histogram(self):
'''Memory pool fee histogram.
TODO: The server should detect and discount transactions that
never get mined when they should.
return self.mempool.get_fee_histogram()
async def relayfee(self):
'''The minimum fee a low-priority tx must pay in order to be accepted
to the daemon's memory pool.'''
......@@ -875,7 +881,7 @@ class Controller(ServerBase):
if not raw_tx:
return None
raw_tx = util.hex_to_bytes(raw_tx)
tx, tx_hash = self.coin.DESERIALIZER(raw_tx).read_tx()
tx = self.coin.DESERIALIZER(raw_tx).read_tx()
if index >= len(tx.outputs):
return None
return self.coin.address_from_script(tx.outputs[index].pk_script)
......@@ -68,10 +68,6 @@ class Env(EnvBase):
self.bandwidth_limit = self.integer('BANDWIDTH_LIMIT', 2000000)
self.session_timeout = self.integer('SESSION_TIMEOUT', 600)
self.irc = self.boolean('IRC', False)
self.irc_nick = self.default('IRC_NICK', None)
# Identities
clearnet_identity = self.clearnet_identity()
tor_identity = self.tor_identity(clearnet_identity)
......@@ -104,7 +100,7 @@ class Env(EnvBase):
or host.lower() == 'localhost')
bad = (ip.is_multicast or ip.is_unspecified
or (ip.is_private and (self.irc or self.peer_announce)))
or (ip.is_private and self.peer_announce))
if bad:
raise self.Error('"{}" is not a valid REPORT_HOST'.format(host))
tcp_port = self.integer('REPORT_TCP_PORT', self.tcp_port) or None
# Copyright (c) 2016-2017, Neil Booth
# All rights reserved.
# See the file "LICENCE" for information about the copyright
# and warranty status of this software.
'''IRC connectivity to discover peers.
Only calling start() requires the IRC Python module.