pickle puzzle

[chan] bitmessage
Feb 11 18:49

I tried to unpickle knownnodes.dat, remove all the non-onion entries, then repickle it so that bitmessage will load it properly. I tried with standard pickle commands, edited the json output, but when I repickle it for some reason bitmessage does not recognize it and just erases the file and starts over on startup with the default bootstrap entries. Bitmessage is not able to load the edited pickle. Does any python guru care to lend a hand at what I need to investigate to get the desired result?

[chan] bitmessage
Feb 11 18:59

for what?

[chan] bitmessage
Feb 11 19:17

the original post already answered this question

[chan] bitmessage
Feb 11 19:29

It didn't answer WHY you want to remove the non-onion entries. Especially since BitMessage is just going to get them from the connected nodes again anyway.

[chan] bitmessage
Feb 11 19:36

Exactly. Why someone wants to delete non-onion entries?

[chan] bitmessage <<Ext>>
Feb 11 23:34

Add to torrc: SocksPort 127.0.0.1:9151 IsolateDestAddr OnionTrafficOnly Then configure bitmessage to use that new port 9151 and it will only connect to onion nodes. Optionally you can delete knownnodes.dat and add to keys.dat: [knownnodes] maxnodes = 1000 This will reduce the number of nodes in knownnodes to 1000, retreive the freshest onion nodes, and connect to those onion nodes them faster. Don't set maxnodes too low or you may never get more than the default boostrap onion node. If you lower maxnodes and don't delete knownnodes.dat, you may end up with 1000 non-onion nodes and never connect at all.

[chan] bitmessage <<Ext>>
Feb 12 00:12

import argparse import imp import sys try: import cPickle as pickle except ImportError: import pickle # Dynamically generate and import stripped down 'state' module containing just # the Peer namedtuple to unpickle knownnodes successfully and output a pickle # data stream PyBitmessage understands module_name = 'state' module_code = '''from collections import namedtuple Peer = namedtuple('Peer', ('host', 'port')) ''' module = imp.new_module(module_name) exec module_code in module.__dict__ sys.modules[module_name] = module parser = argparse.ArgumentParser() parser.add_argument('knownnodes', type=argparse.FileType('rb')) args = parser.parse_args() streams = {} for stream, nodes in pickle.load(args.knownnodes).iteritems(): streams[stream] = {peer: attrs for peer, attrs in nodes.iteritems() if peer.host[-6:] == '.onion'} pickle.dump(streams, sys.stdout)

[chan] bitmessage
Feb 12 04:19

In much older threads, some have worried that malware could overwrite knownnodes.dat with malicious bootstrap entries. I don't recall the results of the threads, but erasing the file and starting over with the default bootstrap entries would be a good precaution for bitmessage to perform in that situation.

[chan] bitmessage <<Ext>>
Feb 12 05:16

If malware is able to modify knownnodes.dat, I'd be more worried about an exploit than malicious bootstrap nodes.

[chan] bitmessage
Feb 12 06:29

WHY? none of your business, that's why.

[chan] bitmessage
Feb 12 08:16

Add to torrc: SocksPort 127.0.0.1:9151 IsolateDestAddr OnionTrafficOnly Then configure bitmessage to use that new port 9151 and it will only connect to onion nodes. Optionally you can delete keys.dat and add to knownnodes.dat: [knownnodes] maxnodes = 1000 This will reduce the number of nodes in knownnodes to 1000, retreive the freshest onion nodes, and connect to those onion nodes them faster. Don't set maxnodes too low or you may never get more than the default boostrap onion node. If you lower maxnodes and don't delete keys.dat, you may end up with 1000 non-onion nodes and never connect at all.

[chan] bitmessage
Feb 12 11:02

I rate it an eight, mate.

[chan] bitmessage
Feb 12 11:03

gr8 b8, m8, I r8 8/8

[chan] bitmessage
Feb 12 16:15

You think you have superior rationale. The subject is not malware. The subject is how to load the pickle with onion only nodes. It doesn't matter why.

[chan] bitmessage
Feb 12 16:15

If I were malware I would send me your keys.

[chan] bitmessage
Feb 12 16:15

this is a clever workaround

[chan] bitmessage
Feb 12 16:18

> exec module_code in module.__dict__ dangerous vector for injection

[chan] bitmessage <<Ext>>
Feb 12 21:23

Not really. An attacker has to be able to modify memory or the file. exec or no exec, an attacker can do whatever they want if they can modify memory or the file.

[chan] bitmessage <<Ext>>
Feb 13 00:04

from argparse import ArgumentParser, FileType from collections import namedtuple from copy_reg import _reconstructor Peer = namedtuple('Peer', ('host', 'port')) safe_class_map = { 'copy_reg._reconstructor': _reconstructor, '__builtin__.tuple': tuple, 'state.Peer': Peer, } def get_class(module, name): fullname = '%s.%s' % (module, name) klass = safe_class_map.get(fullname, None) if not klass: raise UnpicklingError('Unpickling %s not supported' % fullname) return klass parser = ArgumentParser(description='Remove non-onion entries from knownnodes') parser.add_argument('input', nargs='?', default='-', type=FileType('rb'), help='Path of knownnodes.dat to filter or - for stdin (default: stdin)', metavar='in_knownnodes') parser.add_argument('output', nargs='?', default='-', type=FileType('wb'), help='Path to save filtered knownnodes or - for stdout (default: stdout)', metavar='out_knownnodes') args = parser.parse_args() try: import cPickle as pickle unpickler = pickle.Unpickler(args.input) unpickler.find_global = get_class except ImportError: import pickle class Unpickler(pickle.Unpickler): def find_class(self, module, name): return get_class(module, name) unpickler = Unpickler(args.input) streams = {} for stream, nodes in unpickler.load().iteritems(): streams[stream] = {peer: attrs for peer, attrs in nodes.iteritems() if peer.host[-6:] == '.onion'} args.output.write(pickle.dumps(streams).replace('__main__', 'state', 1))

[chan] bitmessage
Feb 13 01:03

slick work

[chan] bitmessage
BM-2cWy7cvHoq3f1rYMerRJp8PT653jjSuEdY

Subject Last Count
bliss Feb 25 09:49 6
no TLS Feb 25 09:01 2
How to start an argument in geekspace Feb 25 08:43 43
Anonymity improvement idea for Bitmessage Feb 25 07:49 8
interface improvement Feb 25 02:05 1
Abit 1.0-rc1 Feb 24 18:18 2
http://33xtkivab2nthghe.onion/7uim34gdxs5z6b5l72nbji7ste Feb 24 08:36 1
Bitmessage security suggestion Feb 24 04:01 27
Fixes #1131 -- typo corrected Feb 23 22:19 1
little fish Feb 23 20:07 1
test Feb 23 17:43 3
Fixes #1134 Feb 23 14:42 2
Fixes #1131 Feb 23 11:37 1
Bitmessage feature request for API commands Feb 23 01:19 10
bitmessage launches cmd and then powershell Feb 22 15:53 56
bitmessage tor service Feb 22 13:31 6
I want the FEDS on this chan to know I identified one of their new tactics. Feb 20 12:03 2
Mitigating exploited software with firejail Feb 19 22:42 8
Critical vulnerability in v0.6.2 Feb 19 16:51 50
message database seems to be corrupted after all that upgraes and attacks Feb 19 14:55 7
Since upgrading yesterday to 6.3.2, Bitmessage is not connecting Feb 19 11:12 7
Inflood of old messages Feb 18 19:16 23
It is slow making connection. Feb 18 18:04 1
Globewashing Feb 18 17:44 1
how to make bitmessage secure Feb 18 05:02 1
Are you blacklisted/whitelisted? Feb 18 04:19 2
Are Linux systems vulnerable to recent attack? Feb 18 02:19 12
Are you blacklisted? Feb 18 02:09 1
address on Peter's reddit account Feb 17 23:51 3
Can't add entries to black list using Add Entry button Feb 17 15:20 4
Errors while trying to run 0.6.2 or 0.6.1 Feb 17 15:20 4
Bitmessage project looking for auditors and/or security specialists (reddit crosspost) Feb 17 13:21 6
HIRE A HACKER/CHANGE GRADES Feb 17 08:59 2
Download it. Feb 17 07:59 2
passphrase strength ? Feb 16 20:34 8
$ cd PyBitmessage ; git log | grep Author | sort -u | blacklist Feb 16 15:54 18
diagram Feb 16 01:46 1
Bitmessage components security seclusion example Feb 16 01:24 1
โฉฉ ๐„‰ ใŽฎ ไทฆ ๐Ÿžณ ๐Ÿ† ใ ๏‡บ f ๐™ฒ ๐Ÿ„ฆ โž‡ โจ˜ ใŠณ ๐— โฆฑ ๓ฟฟป ๏ค ๐Ÿ„น ๐Ÿ’ ๎… ไท„ ๎Žพ ๏ผซ ๎ƒ— ๐Ÿ†™ ๏€• ไทค ๐™ ๎Œฅ โ’„ โ‚น ๊ ฒ ๎‘• Feb 16 00:04 1
NOTICE: Address Revocation Feb 15 18:28 12
Cannot connect since yesterday Feb 15 17:59 2
Questions regarding recent bitmessage data exploit Feb 15 03:46 2
Latest commit borked Feb 14 05:26 5
BM-onion Feb 14 05:22 5
That's my new address Feb 14 03:40 1
BM massacre! Feb 13 21:23 2
Namecoin integration Feb 13 20:18 11
Hashwalling Functions for Security Feb 13 17:58 2
Same old problem connecting to network Feb 13 17:12 4
Injection attack mitigation Feb 13 16:52 7
This denial of service shit needs to be patched Feb 13 12:00 7
Test Feb 13 11:37 1
Proving that BM was sent? Feb 13 11:07 10
bitmessage ... Feb 13 08:13 1
Improve icon for chan + messages: important or not Feb 13 05:25 2
pickle puzzle Feb 13 01:03 20
so happy Feb 12 16:32 2
Fwd: Re: Did everyone else's BM starting freezing up Feb 11 03:54 10
hacker service Feb 10 03:48 2
another feature request Feb 10 01:12 1
bitmessage feature request Feb 10 01:10 1
feature request Feb 10 01:04 1
Questions for the Bitmessage Community Feb 9 21:30 7
Did everyone else's BM starting freezing up Feb 9 03:21 4
A light weight version of the denial of service message Feb 8 13:22 3
RE: Hello. Feb 8 11:48 1
WWtest Feb 8 10:44 1
test1 Feb 8 10:37 1
WARNING! denial of service message Feb 8 10:19 3
extended encoding Feb 8 01:24 7
bountyfy -- 7 โ‚ฌ payout Feb 5 20:59 2
clean up pyBM github landing page, please Feb 4 23:00 2
Running BM daemon as a service Feb 4 13:47 6
hidden service - long names Feb 4 12:37 7
RAM consumption - RAM not released Feb 3 21:05 4
Bug? First connection quickly breaks Feb 3 11:41 6
Request: debug.log initialization / termination Feb 2 18:30 2
kqueue poller in asyncore bounty -- no payout Feb 2 14:23 5
Bitmessage bug in Help > About Feb 2 13:59 7
Message size is metadata Feb 2 13:25 6
New warning "sni-qt/5864" WARN Feb 2 12:12 2
ordering Feb 1 10:38 12
RAM consumption Feb 1 10:14 5
discrepancy in transmit/receive byte counts Feb 1 07:53 6
BM CPU time Feb 1 02:39 5
kqueue poller in asyncore bounty Feb 1 00:13 15
new theme for beamstat Jan 31 11:35 2
Support request -- dontconnect in pyBM 062 not being honoured Jan 31 10:16 1
python IDE Jan 31 10:15 2
My BM is connected to one peer twice Jan 30 06:36 7
Support request/Bug report: keys.dat gets corrupted when running out of disk space Jan 29 15:44 2
Feature request/idea/suggestion: user-defined data directory (command-line argument) Jan 29 15:16 2
GUI dontsendack Jan 29 05:15 1
Another message problem Jan 29 03:49 3
Message deletion broken Jan 29 00:28 3
bitmessage on android device Jan 29 00:03 1